1
DNS Anomalies and
Their Impacts on
DNS Cache Servers
Katsuyasu TOYAMA, Keisuke ISHIBASHI, and
Tsuyoshi TOYONO (NTT)
Masa...
2
Background
In the DNS world, these ‘players’ are related to
each other
1. application or operating system in user device...
3
Today’s topic
The burden of DNS Cache servers caused by
misconfigured DNS authoritative servers
Lame delegation is well ...
4
1. Virus and Worm
5
Activities of Virus and Worm
Activities of viruses and worms sometimes
cause burden on DNS cache servers.
MyDoom
Attacks...
6
What is Antinny?
Antinny is a worm that infects through famous
Japanese P2P software, “Winny”.
Some subspecies of Antinn...
7
Worms’ impacts on networks
The server is
under attack!!
Target website
ISP
Worm-infected users Worm-infected user
DDoS a...
8
Owner removed A RR from
authoritative server!!
Web server
DNS cache
servers
Worm-infected user
DNS authoritative server
...
9
Effect of A RR removal
Web site owner is happy because:
it is very easy to remove A RR
their link has become very quiet
...
10
Why is DNS cache overloaded?
‘A RR’ was removed from authoritative server.
NXDOMAIN was returned, and its TTL was short...
11
No iterative query
Quick fix: Return blackhole IP
address by each cache server
HTTP Server
DNS Authoritative
Server
Esc...
12
Increased total queries at DNS cache
DDoS began, then
A RR was removed
Blackhole address
was returned
Apr. 5, Monday
An...
13
Number of queries resolving FQDN of target
Web server (May 1st – 7th, per 5 min.)
1
10
100
1000
10000
100000
1000000
5/...
14
Number of queries resolving FQDN of target
Web server (May 3rd, 01:00 – 03:00, per seconds)
0
500
1000
1500
2000
2500
3...
15
Number of unique IP address
(May 3rd, 01:00 – 03:00, per second)
# of unique IP address increased only 2-3 times
in “NX...
16
Solution:
Return Blackhole IP Address by authoritative server
Web server
DNS cache
servers
No
heavy load
Blackhole
Targ...
17
Lessons from the attack
Removing A RR is not a good method
Behavior of worms, resolvers at user, and
authoritative serv...
18
2. Large RRSet and TCP filtering
19
Overload of DNS cache server
Increasing load of DNS cache server, but..
Number of total queries was normal
We detected ...
20
Why TCP queries were increased
The number of truncated answers and that of
TCP packets are synchronized
We guessed trun...
21
Why TCP queries increased
(cont’d)
What happened was…
(1) The cache server sent iterative query to
the authoritative se...
22
Dump example
% dig @yyy.yyy.yyy.yyy -x zzz.zzz.zzz.zzz
21:23:14.604332 IP xxx.xxx.xxx.xxx.51705 > yyy.yyy.yyy.yyy.53: 2...
23
Workaround
1. Asked the administrators of the authoritative
server
to change the settings to accept TCP connection, or
...
24
Another possible approach
Patching BIND, not to query by TCP when
truncation occurs.
Quick hacking could reduce TCP ses...
25
Lessons through this phenomenon
Strongly recommend that administrators
check the configuration of authoritative
servers...
26
Summary
To be ‘friendly’ to DNS cache servers,
Administrators should check and modify
the settings of authoritative ser...
27
Contacts
Chika YOSHIMURA (NTT Communications)
yosimura@ocn.ad.jp
Katsuyasu TOYAMA (NTT)
toyama.katsuyasu@lab.ntt.co.jp
28
Special Thanks to:
Ichiro Mizukoshi (NTT Communications)
Haruhiko Ohshima (NTT Communications)
Yasuhiro Morishita (JPRS...
of 28

NANOG32 - DNS Anomalies and Their Impacts on DNS Cache Servers

NANOG32 - DNS Anomalies and Their Impacts on DNS Cache Servers
Published on: Mar 3, 2016
Published in: Internet      
Source: www.slideshare.net


Transcripts - NANOG32 - DNS Anomalies and Their Impacts on DNS Cache Servers

  • 1. 1 DNS Anomalies and Their Impacts on DNS Cache Servers Katsuyasu TOYAMA, Keisuke ISHIBASHI, and Tsuyoshi TOYONO (NTT) Masahiro ISHINO and Chika YOSHIMURA (NTT Communications) Kazunori FUJIWARA (JPRS)
  • 2. 2 Background In the DNS world, these ‘players’ are related to each other 1. application or operating system in user devices 2. authoritative servers including root DNS servers 3. DNS cache servers DNS Cache servers Authoritative servers example. com .com “.” User
  • 3. 3 Today’s topic The burden of DNS Cache servers caused by misconfigured DNS authoritative servers Lame delegation is well known, but other factors exist as well. Focusing on: 1. Virus and Worm activities An extreme increase in queries caused DNS cache server to become overloaded. 2. Large RRSet and TCP filtering Even small number of queries could cause increase in heavy load of DNS cache server.
  • 4. 4 1. Virus and Worm
  • 5. 5 Activities of Virus and Worm Activities of viruses and worms sometimes cause burden on DNS cache servers. MyDoom Attacks SCO Web site; some subspecies attack Microsoft Web site Antinny Attacks ACCSJP Web site ACCSJP: Association of Copyright for Computer Software
  • 6. 6 What is Antinny? Antinny is a worm that infects through famous Japanese P2P software, “Winny”. Some subspecies of Antinny try to send private information of the infected user to the Web site of ACCSJP once a month. the first Monday of the month Apr. 5th, May 3rd, June 7th… to expose the user as a potential criminal?? When trying to connect to the Web site, it resolves the FQDN “www.accsjp.or.jp”
  • 7. 7 Worms’ impacts on networks The server is under attack!! Target website ISP Worm-infected users Worm-infected user DDoS attack DDoS attack Backbone is also filled with DDoS packets Access line is filled with DDoS packets
  • 8. 8 Owner removed A RR from authoritative server!! Web server DNS cache servers Worm-infected user DNS authoritative server (1) Worms intensively repeat sending queries to resolve the FQDN. (2) Iterative queries immediately after TTL expires (3) No such record (NXDOMAIN) (4) “NXDOMAIN” Heavy load 6 times or more DDoS attack Escape from DDoS attack Target website
  • 9. 9 Effect of A RR removal Web site owner is happy because: it is very easy to remove A RR their link has become very quiet But… DNS cache servers are overloaded!
  • 10. 10 Why is DNS cache overloaded? ‘A RR’ was removed from authoritative server. NXDOMAIN was returned, and its TTL was short (60 sec.) Worms repeatedly sent queries even if they could not resolve the name. They never gave up… The highest was approximately 700 queries per second from an IP address! Negative cache did not seem to be effective in some Operating Systems or applications Negative cache is disabled at any time?
  • 11. 11 No iterative query Quick fix: Return blackhole IP address by each cache server HTTP Server DNS Authoritative Server Escape from DDoS attack DNS cache servers (Fake) zone forwarders (2) Blackhole IP address (1) Worms send queries to resolve the FQDN. No heavy load Blackhole Target website Worm-infected user (3) DDoS attack (4) Discard
  • 12. 12 Increased total queries at DNS cache DDoS began, then A RR was removed Blackhole address was returned Apr. 5, Monday Antinny activated May. 3, Monday Antinny activated Green: Number of total queries Blue: Number of NXDOMAIN
  • 13. 13 Number of queries resolving FQDN of target Web server (May 1st – 7th, per 5 min.) 1 10 100 1000 10000 100000 1000000 5/1 5/2 5/3 5/4 5/5 5/6 5/7 5/8 #ofqueries(5min) ▲ Monday May 3 Much more queries than usual This spike indicates extremely large number of queries
  • 14. 14 Number of queries resolving FQDN of target Web server (May 3rd, 01:00 – 03:00, per seconds) 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 0:00:00 1:00:00 2:00:00 3:00:00 #ofqueris(1second) Queries extremely increased only when NXDOMAIN returned Every 60 seconds, queries increased Every 600 seconds, queries increased but not so many Spikes when NXDOMAIN returned Original A RR with TTL 60 sec Blackhole address with TTL 600 sec NXDomain with TTL 60 sec Changes in answers
  • 15. 15 Number of unique IP address (May 3rd, 01:00 – 03:00, per second) # of unique IP address increased only 2-3 times in “NXDomain period”, while queries increased 10 times or more! 0 10 20 30 40 50 60 0:00:00 1:00:00 2:00:00 3:00:00 #ofuniqeusers(persec) Original A RR with TTL 60 sec Blackhole address with TTL 600 sec NXDomain with TTL 60 sec Changes in answers
  • 16. 16 Solution: Return Blackhole IP Address by authoritative server Web server DNS cache servers No heavy load Blackhole Target website Worm-infected user (5) DDoS attack Return Blackhole IP address (4) Blackhole IP address (2) Iterative queries DNS Authoritative Server (3) Blackhole IP address (1) Worms send queries to resolve the FQDN. In June, ISPs collaborated to defend the attack Asked the administrator of the authoritative server to return a blackhole address Escape from DDoS attack (6) Discard
  • 17. 17 Lessons from the attack Removing A RR is not a good method Behavior of worms, resolvers at user, and authoritative servers sometimes causes DNS cache server to collapse. Collaboration between victims and ISPs is required. This time blackhole IP address worked very well. Generic blackhole addresses are necessary If ISPs set the blackhole addresses to be discarded at their routers, then an authoritative server can return the addresses to escape from DDoS attacks. For instance, TEST-NET (192.0.2.0/24, RFC3330)
  • 18. 18 2. Large RRSet and TCP filtering
  • 19. 19 Overload of DNS cache server Increasing load of DNS cache server, but.. Number of total queries was normal We detected increase in the number of TCP queries 0 20 40 60 80 100 120 140 6/8 0:00 6/8 6:00 6/8 12:00 6/8 18:00 6/9 0:00 6/9 6:00 CPU Utilization of Named(%) Numbers of SYN_SENT for DNS
  • 20. 20 Why TCP queries were increased The number of truncated answers and that of TCP packets are synchronized We guessed truncated answers caused TCP queries. Number of truncated answers from authoritative servers (per 5 min.) Number of TCP packets sent to authoritative servers (per 5 min.) 0 100 200 300 400 500 600 6/5 6/6 6/7 6/8 0 100 200 300 400 500 600 6/5 6/6 6/7 6/8
  • 21. 21 Why TCP queries increased (cont’d) What happened was… (1) The cache server sent iterative query to the authoritative servers. (2) The authoritative servers answered with a truncated bit; over 512 octets answer (3) Then the cache server tried to resend query by TCP. The authoritative servers did not respond to TCP query. The authoritative servers didn’t support EDNS0 TCP packets seemed to be filtered before authoritative server process The cache server was waiting 75 seconds for acknowledgement from the authoritative servers, and pending queries were increased. ⇒ the cache server overloaded ServFail (did not cache the result) Cache server User query Timeout Authoritative servers (1)Iterative query (2)Answer with TC bit (3)Resend query by TCP (3)Retry No response No response (4)Try another (UDP,TCP) No response
  • 22. 22 Dump example % dig @yyy.yyy.yyy.yyy -x zzz.zzz.zzz.zzz 21:23:14.604332 IP xxx.xxx.xxx.xxx.51705 > yyy.yyy.yyy.yyy.53: 24775+ PTR? zzz.zzz.zzz.zzz.in-addr.arpa. (42) 21:23:14.744362 IP yyy.yyy.yyy.yyy.53 > xxx.xxx.xxx.xxx.51705: 24775| 15/0/0 PTR redirect.***.co.in., PTR onlyoriginal.com.my., PTR redirect.***.com.cn., PTR redirect.china.***.com., PTR redirect.jp.***.com., PTR ***.co.jp., PTR redirect.***.co.jp., PTR ***.ne.jp., PTR redirect.***.co.kr., PTR redirect.kt***.co.kr., PTR redirect.***.com.my., PTR redirect.***.com.ph., PTR redirect.***.com.sg., PTR redirect.***.co.th., PTR redirect.***.com.tw. (488) 21:23:14.744748 IP xxx.xxx.xxx.xxx.60035 > yyy.yyy.yyy.yyy.53: S 2600114847:2600114847(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,timestamp 1455660 0> 21:23:17.736935 IP xxx.xxx.xxx.xxx.60035 > yyy.yyy.yyy.yyy.53: S 2600114847:2600114847(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,timestamp 1455960 0> 21:23:20.937001 IP xxx.xxx.xxx.xxx.60035 > yyy.yyy.yyy.yyy.53: S 2600114847:2600114847(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,timestamp 1456280 0> TC bit (1) Iterative query (2) Answer with TC bit (3) Resend query by TCP (SYN packet)
  • 23. 23 Workaround 1. Asked the administrators of the authoritative server to change the settings to accept TCP connection, or to decrease the size of their records to fit them into a UDP packet. --> But the administrators will not change the settings… 2. Denied the users sending the query by using BIND blackhole setting
  • 24. 24 Another possible approach Patching BIND, not to query by TCP when truncation occurs. Quick hacking could reduce TCP sessions, but it may violate RFC. Proper modification would be: Cache the following information for the RR during the TTL TC = 1 Does not accept TCP query Return ServFail without iterative query if above cache exists …this should be proposed to IETF
  • 25. 25 Lessons through this phenomenon Strongly recommend that administrators check the configuration of authoritative servers. Answer TCP queries (mandatory) RFC 1123 DNS servers must be able to service UDP and should be able to service TCP queries … it should not refuse to service a TCP query just because it would have succeeded with UDP. and either Set the size of answer to be smaller than 512 octets, or Support EDNS0 option
  • 26. 26 Summary To be ‘friendly’ to DNS cache servers, Administrators should check and modify the settings of authoritative servers. Some generic blackhole address should be used, instead of removing A RR in case of DDoS. Configuration should be consistent. Oversized RRSet with no EDNS0 and closed TCP port is not good but often seen.
  • 27. 27 Contacts Chika YOSHIMURA (NTT Communications) yosimura@ocn.ad.jp Katsuyasu TOYAMA (NTT) toyama.katsuyasu@lab.ntt.co.jp
  • 28. 28 Special Thanks to: Ichiro Mizukoshi (NTT Communications) Haruhiko Ohshima (NTT Communications) Yasuhiro Morishita (JPRS) Hirotaka Matsuoka (NTT)

Related Documents