도커 네트워크
앞의 컨테이너 격리에 이어서
Kubernetes Advanced Networking Study(이하 KANG)첫번째 시간에 내용을 정리 하였습니다. 컨테이너 격리를 아직 보시지 않으셨다면 보시고 읽기를 권장해드립니다.
실습 환경 구성
새로운 빈 디렉토리를 생성하고 Vagrantfile를 아래와 같은 내용으로 생성합니다.
Vagrant.configure("2") do |config|
config.vm.boot_timeout = 1800
config.vm.box = "ubuntu/focal64"
config.vm.box_version = "20211026.0.0"
config.vm.provision "shell", inline: "echo 'sudo su -' >> .bashrc"
config.vm.provision "shell", inline: "apt update"
config.vm.provision "shell", inline: "apt install -y conntrack bridge-utils net-tools jq resolvconf tree iptraf-ng"
config.vm.provision "shell", inline: "echo 'nameserver 1.1.1.1' > /etc/resolvconf/resolv.conf.d/head"
config.vm.provision "shell", inline: "resolvconf -u"
config.vm.provision "shell", inline: "curl -fsSL https://get.docker.com | sh"
config.vm.synced_folder "./", "/vagrant", disabled: true
config.vm.define "vm1" do |vm1|
vm1.vm.hostname = "docker1"
vm1.vm.network "private_network", ip: "192.168.50.10"
vm1.vm.network "forwarded_port", guest: 22, host: 60610, auto_correct: true, id: "ssh"
vm1.vm.provider :virtualbox do |spec|
spec.cpus = 2
spec.memory = 1024
spec.name = "docker1"
spec.linked_clone = true
spec.customize [ "modifyvm", :id, "--uartmode1", "disconnected" ]
# 어댑터 2에 모두 허용 처리
spec.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
end
end
config.vm.define "vm2" do |vm2|
vm2.vm.hostname = "docker2"
vm2.vm.network "private_network", ip: "192.168.50.20"
vm2.vm.network "forwarded_port", guest: 22, host: 60620, auto_correct: true, id: "ssh"
vm2.vm.provider :virtualbox do |spec|
spec.cpus = 2
spec.memory = 1024
spec.name = "docker2"
spec.linked_clone = true
spec.customize [ "modifyvm", :id, "--uartmode1", "disconnected" ]
# 어댑터 2에 모두 허용 처리
spec.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
end
end
config.vm.define "vm3" do |vm3|
vm3.vm.hostname = "gw"
vm3.vm.network "private_network", ip: "192.168.50.254"
vm3.vm.network "forwarded_port", guest: 22, host: 60630, auto_correct: true, id: "ssh"
vm3.vm.provider :virtualbox do |spec|
spec.cpus = 2
spec.memory = 1024
spec.name = "gw"
spec.linked_clone = true
spec.customize [ "modifyvm", :id, "--uartmode1", "disconnected" ]
# 어댑터 2에 모두 허용 처리
spec.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
end
end
end
- 추가로 Virtualbox 설정 : 노드(VM)에 네트워크 NIC 2 번에
무작위 모드를모두 허용설정이 Vagrantfile 에 의해 되어 있습니다. 모두 허용일 경우패킷 스니핑(Sniffing)이 가능하여wireshark등을 사용을 할 수 있습니다.
vagrant 명령어로 Ubuntu 3대 생성을 합니다.
$ vagrant up
# 배포 상태 확인
$ vagrant status
Current machine states:
vm1 running (virtualbox)
vm2 running (virtualbox)
vm3 running (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
# 터미널1 (enp0s8 - 192.168.50.10)
$ vagrant ssh vm1
# 터미널2 (enp0s8 - 192.168.50.20)
$ vagrant ssh vm2
# 터미널3 (enp0s8 - 192.168.50.254)
$ vagrant ssh vm3
RED <==> BLUE 네트워크 네임스페이스 간 통신
- RED와 BLUE 네트워크 네임스페이스를 생성하고 직접 연결하여 통신을 테스트 합니다.
veth 에서 v 는 virtual 을 뜻 합니다. 즉 virtual ethernet 라는 의미입니다.
네트워크 네임스페이스를 생성하면 Host 네트워크와 구별되어 생성됩니다.
# veth (가상 이더넷 디바이스) 생성, man ip-link
$ ip link add veth0 type veth peer name veth1
# veth 생성 확인(상태 DOWN), ifconfig 에는 peer 정보 확인 안됨
$ ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 08:00:27:66:f0:26 brd ff:ff:ff:ff:ff:ff
4: veth1@veth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 82:42:52:ac:5b:49 brd ff:ff:ff:ff:ff:ff
5: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether ea:ad:dd:3b:ca:20 brd ff:ff:ff:ff:ff:ff
$ ip -c addr # 축약 ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 85943sec preferred_lft 85943sec
inet6 fe80::31:14ff:fe45:3203/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 08:00:27:66:f0:26 brd ff:ff:ff:ff:ff:ff
4: veth1@veth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 82:42:52:ac:5b:49 brd ff:ff:ff:ff:ff:ff
5: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ea:ad:dd:3b:ca:20 brd ff:ff:ff:ff:ff:ff
# 네트워크 네임스페이스 생성 , man ip-netns
$ ip netns add RED
$ ip netns add BLUE
# 네트워크 네임스페이스 확인
$ ip netns list
BLUE
RED
# veth0 을 RED 네트워크 네임스페이스로 옮김
$ ip link set veth0 netns RED
$ ip netns list
BLUE
RED (id: 0)
## 호스트의 ip a 목록에서 보이지 않음, veth1의 peer 정보가 변경됨
$ ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 08:00:27:66:f0:26 brd ff:ff:ff:ff:ff:ff
4: veth1@if5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 82:42:52:ac:5b:49 brd ff:ff:ff:ff:ff:ff link-netns RED
## RED 네임스페이스에서 ip a 확인됨(상태 DOWN), peer 정보 확인, link-netns RED, man ip-netns
$ ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
5: veth0@if4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ea:ad:dd:3b:ca:20 brd ff:ff:ff:ff:ff:ff link-netnsid 0
# veth1 을 BLUE 네트워크 네임스페이스로 옮김
$ ip link set veth1 netns BLUE
$ ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 08:00:27:66:f0:26 brd ff:ff:ff:ff:ff:ff
$ ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: veth1@if7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ca:0d:cb:43:e8:9b brd ff:ff:ff:ff:ff:ff link-netns RED
# veth0, veth1 상태 활성화(state UP)
$ ip netns exec RED ip link set veth0 up
$ ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
7: veth0@if6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
link/ether 66:6f:02:41:7d:1f brd ff:ff:ff:ff:ff:ff link-netns BLUE
$ ip netns exec BLUE ip link set veth1 up
$ ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: veth1@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ca:0d:cb:43:e8:9b brd ff:ff:ff:ff:ff:ff link-netns RED
inet6 fe80::c80d:cbff:fe43:e89b/64 scope link tentative
valid_lft forever preferred_lft forever
# veth0, veth1 에 IP 설정
$ ip netns exec RED ip addr add 11.11.11.2/24 dev veth0
$ ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
7: veth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 66:6f:02:41:7d:1f brd ff:ff:ff:ff:ff:ff link-netns BLUE
inet 11.11.11.2/24 scope global veth0
valid_lft forever preferred_lft forever
inet6 fe80::646f:2ff:fe41:7d1f/64 scope link
$ ip netns exec BLUE ip addr add 11.11.11.3/24 dev veth1
$ ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: veth1@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ca:0d:cb:43:e8:9b brd ff:ff:ff:ff:ff:ff link-netns RED
inet 11.11.11.3/24 scope global veth1
valid_lft forever preferred_lft forever
inet6 fe80::c80d:cbff:fe43:e89b/64 scope link
valid_lft forever preferred_lft forever
# 터미널1 (RED 11.11.11.2)
## nsenter : 네임스페이스에 attach 하여 지정한 프로그램을 실행
$ nsenter --net=/var/run/netns/RED
$ ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
7: veth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 66:6f:02:41:7d:1f brd ff:ff:ff:ff:ff:ff link-netns BLUE
inet 11.11.11.2/24 scope global veth0
valid_lft forever preferred_lft forever
inet6 fe80::646f:2ff:fe41:7d1f/64 scope link
valid_lft forever preferred_lft forever
## neighbour/arp tables management , man ip-neighbour
$ ip -c neigh
## 라우팅 정보, iptables 정보
$ ip -c route
11.11.11.0/24 dev veth0 proto kernel scope link src 11.11.11.2
$ iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
$ iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
# 터미널2 (호스트)
$ lsns -t net # nsenter 실행 후 TYPE(net) CMD(-bash) 생성 확인
NS TYPE NPROCS PID USER NETNSID NSFS COMMAND
4026531992 net 113 1 root unassigned /sbin/init
4026532180 net 1 3137 root 0 /run/netns/RED -bash
$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 84535sec preferred_lft 84535sec
inet6 fe80::31:14ff:fe45:3203/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 08:00:27:66:f0:26 brd ff:ff:ff:ff:ff:ff
$ ip -c neigh
10.0.2.2 dev enp0s3 lladdr 52:54:00:12:35:02 DELAY
10.0.2.3 dev enp0s3 lladdr 52:54:00:12:35:03 STALE
$ ip -c route
default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
$ iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
$ iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
# 터미널3 (BLUE 11.11.11.3)
$ nsenter --net=/var/run/netns/BLUE
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: veth1@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ca:0d:cb:43:e8:9b brd ff:ff:ff:ff:ff:ff link-netns RED
inet 11.11.11.3/24 scope global veth1
valid_lft forever preferred_lft forever
inet6 fe80::c80d:cbff:fe43:e89b/64 scope link
valid_lft forever preferred_lft forever
$ ip -c neigh
$ ip -c route
11.11.11.0/24 dev veth1 proto kernel scope link src 11.11.11.3
$ iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
$ iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
# ping 통신 확인
# 터미널3 (BLUE 11.11.11.3)
$ tcpdump -i veth1
# ping을 호출 후 아래 내용 출력됨
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth1, link-type EN10MB (Ethernet), capture size 262144 bytes
06:09:18.956745 ARP, Request who-has 11.11.11.3 tell 11.11.11.2, length 28
06:09:18.956760 ARP, Reply 11.11.11.3 is-at ca:0d:cb:43:e8:9b (oui Unknown), length 28
06:09:18.956762 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 3696, seq 1, length 64
06:09:18.956772 IP 11.11.11.3 > 11.11.11.2: ICMP echo reply, id 3696, seq 1, length 64
06:09:24.094659 ARP, Request who-has 11.11.11.2 tell 11.11.11.3, length 28
06:09:24.094865 ARP, Reply 11.11.11.2 is-at 66:6f:02:41:7d:1f (oui Unknown), length 28
$ ip -c neigh
11.11.11.2 dev veth1 lladdr 66:6f:02:41:7d:1f STALE
# 터미널1 (RED 11.11.11.2)
$ ping 11.11.11.3 -c 1
PING 11.11.11.3 (11.11.11.3) 56(84) bytes of data.
64 bytes from 11.11.11.3: icmp_seq=1 ttl=64 time=0.045 ms
--- 11.11.11.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.045/0.045/0.045/0.000 ms
$ ip -c neigh
11.11.11.3 dev veth0 lladdr ca:0d:cb:43:e8:9b STALE
# 삭제 (호스트)
ip netns delete RED
ip netns delete BLUE
RED <== Bridge(br0) ==> BLUE 간 통신
- arp table, route table, iptables 와 호스트의 bridge fdb 를 통하여 통신을 해봅니다.
# 네트워크 네임스페이스 및 veth 생성
$ ip netns add RED
$ ip link add reth0 type veth peer name reth1
$ ip link set reth0 netns RED
$ ip netns add BLUE
$ ip link add beth0 type veth peer name beth1
$ ip link set beth0 netns BLUE
# 확인
$ ip netns list
BLUE (id: 1)
RED (id: 0)
$ ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:40:48:37 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:cd:05:9d:50 brd ff:ff:ff:ff:ff:ff
5: reth1@if6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0e:7f:1d:23:77:01 brd ff:ff:ff:ff:ff:ff link-netns RED
7: beth1@if8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 36:88:d0:d2:f6:03 brd ff:ff:ff:ff:ff:ff link-netns BLUE
$ ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: reth0@if5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 66:f9:63:30:ea:4a brd ff:ff:ff:ff:ff:ff link-netnsid 0
$ ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
8: beth0@if7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 9a:a3:cd:aa:1e:ea brd ff:ff:ff:ff:ff:ff link-netnsid 0
$ ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: reth0@if8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 12:de:f3:9e:68:6a brd ff:ff:ff:ff:ff:ff link-netnsid 0
$ ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
11: beth0@if10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 4a:3e:3b:d5:b9:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0
# 브리지 정보 확인
$ brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.0242cd059d50 no
# br0 브리지 생성
$ ip link add br0 type bridge
# br0 브리지 정보 확인
$ brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.000000000000 no
$ brctl showmacs br0
br0port no mac addr is local? ageing timer
$ brctl showstp br0
br0
bridge id 8000.000000000000
designated root 8000.000000000000
root port 0 path cost 0
max age 20.00 bridge max age 20.00
hello time 2.00 bridge hello time 2.00
forward delay 15.00 bridge forward delay 15.00
ageing time 300.00
hello timer 0.00 tcn timer 0.00
topology change timer 0.00 gc timer 0.00
# reth1 beth1 을 br0 연결
$ ip link set reth1 master br0
$ ip link set beth1 master br0
$ brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.0e7f1d237701 no beth1
reth1
$ brctl showmacs br0
port no mac addr is local? ageing timer
1 0e:7f:1d:23:77:01 yes 0.00
1 0e:7f:1d:23:77:01 yes 0.00
2 36:88:d0:d2:f6:03 yes 0.00
2 36:88:d0:d2:f6:03 yes 0.00
$ ip -br -c link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp0s3 UP 02:31:14:45:32:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
enp0s8 UP 08:00:27:40:48:37 <BROADCAST,MULTICAST,UP,LOWER_UP>
docker0 DOWN 02:42:cd:05:9d:50 <NO-CARRIER,BROADCAST,MULTICAST,UP>
reth1@if6 DOWN 0e:7f:1d:23:77:01 <BROADCAST,MULTICAST>
beth1@if8 DOWN 36:88:d0:d2:f6:03 <BROADCAST,MULTICAST>
br0 DOWN 0e:7f:1d:23:77:01 <BROADCAST,MULTICAST>
# reth0 beth0 에 IP 설정 및 활성화, br0 활성화
$ ip netns exec RED ip addr add 11.11.11.2/24 dev reth0
$ ip netns exec BLUE ip addr add 11.11.11.3/24 dev beth0
$ ip netns exec RED ip link set reth0 up; ip link set reth1 up
$ ip netns exec BLUE ip link set beth0 up; ip link set beth1 up
$ ip link set br0 up
$ ip -br -c addr
ip netns exec RED ip addr add 11.11.11.2/24 dev reth0
ip netns exec BLUE ip addr add 11.11.11.3/24 dev beth0
ip netns exec RED ip link set reth0 up; ip link set reth1 up
ip netns exec BLUE ip link set beth0 up; ip link set beth1 up
ip link set br0 up
ip -br -c addr
# 터미널1 (RED 11.11.11.2)
$ nsenter --net=/var/run/netns/RED
$ ip -c a;echo; ip -c route;echo; ip -c neigh
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: reth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 66:f9:63:30:ea:4a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 11.11.11.2/24 scope global reth0
valid_lft forever preferred_lft forever
inet6 fe80::64f9:63ff:fe30:ea4a/64 scope link
valid_lft forever preferred_lft forever
11.11.11.0/24 dev reth0 proto kernel scope link src 11.11.11.2
## 현재 네트워크 네임스페이스 정보 확인
$ ip netns identify $$
RED
# 터미널2 (호스트)
$ brctl showmacs br0
port no mac addr is local? ageing timer
1 0e:7f:1d:23:77:01 yes 0.00
1 0e:7f:1d:23:77:01 yes 0.00
2 36:88:d0:d2:f6:03 yes 0.00
2 36:88:d0:d2:f6:03 yes 0.00
1 66:f9:63:30:ea:4a no 41.37
2 9a:a3:cd:aa:1e:ea no 27.04
$ bridge fdb show
33:33:00:00:00:01 dev enp0s3 self permanent
01:00:5e:00:00:01 dev enp0s3 self permanent
33:33:ff:45:32:03 dev enp0s3 self permanent
01:80:c2:00:00:00 dev enp0s3 self permanent
01:80:c2:00:00:03 dev enp0s3 self permanent
01:80:c2:00:00:0e dev enp0s3 self permanent
33:33:00:00:00:01 dev enp0s8 self permanent
01:00:5e:00:00:01 dev enp0s8 self permanent
33:33:ff:40:48:37 dev enp0s8 self permanent
01:80:c2:00:00:00 dev enp0s8 self permanent
01:80:c2:00:00:03 dev enp0s8 self permanent
01:80:c2:00:00:0e dev enp0s8 self permanent
33:33:00:00:00:01 dev docker0 self permanent
01:00:5e:00:00:6a dev docker0 self permanent
33:33:00:00:00:6a dev docker0 self permanent
01:00:5e:00:00:01 dev docker0 self permanent
02:42:cd:05:9d:50 dev docker0 vlan 1 master docker0 permanent
02:42:cd:05:9d:50 dev docker0 master docker0 permanent
66:f9:63:30:ea:4a dev reth1 master br0
0e:7f:1d:23:77:01 dev reth1 vlan 1 master br0 permanent
0e:7f:1d:23:77:01 dev reth1 master br0 permanent
33:33:00:00:00:01 dev reth1 self permanent
01:00:5e:00:00:01 dev reth1 self permanent
33:33:ff:23:77:01 dev reth1 self permanent
9a:a3:cd:aa:1e:ea dev beth1 master br0
36:88:d0:d2:f6:03 dev beth1 vlan 1 master br0 permanent
36:88:d0:d2:f6:03 dev beth1 master br0 permanent
33:33:00:00:00:01 dev beth1 self permanent
01:00:5e:00:00:01 dev beth1 self permanent
33:33:ff:d2:f6:03 dev beth1 self permanent
33:33:00:00:00:01 dev br0 self permanent
01:00:5e:00:00:6a dev br0 self permanent
33:33:00:00:00:6a dev br0 self permanent
01:00:5e:00:00:01 dev br0 self permanent
33:33:ff:23:77:01 dev br0 self permanent
$ bridge fdb show dev br0
33:33:00:00:00:01 self permanent
01:00:5e:00:00:6a self permanent
33:33:00:00:00:6a self permanent
01:00:5e:00:00:01 self permanent
33:33:ff:23:77:01 self permanent
$ iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
$ iptables -t filter -L -n -v
Chain INPUT (policy ACCEPT 3471 packets, 8164K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 1545 packets, 74567 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
# 터미널3 (BLUE 11.11.11.3)
$ nsenter --net=/var/run/netns/BLUE
$ ip -c a;echo; ip -c route;echo; ip -c neigh
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
8: beth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 9a:a3:cd:aa:1e:ea brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 11.11.11.3/24 scope global beth0
valid_lft forever preferred_lft forever
inet6 fe80::98a3:cdff:feaa:1eea/64 scope link
valid_lft forever preferred_lft forever
11.11.11.0/24 dev beth0 proto kernel scope link src 11.11.11.3
11.11.11.2 dev beth0 lladdr 66:f9:63:30:ea:4a STALE
## 현재 네트워크 네임스페이스 정보 확인
$ ip netns identify $$
BLUE
==============================================================
# 터미널2 (호스트)
# ping 통신 전 사전 설정
## iptables 정보 확인
$ iptables -t filter -S | grep '\-P'
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
$ iptables -nvL -t filter
Chain INPUT (policy ACCEPT 3543 packets, 8182K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 1607 packets, 80553 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
## Ubuntu 호스트에서 패킷 라우팅 설정 확인 : 커널의 IP Forwarding (routing) 기능 확인 - 0(off), 1(on)
## echo 1 > /proc/sys/net/ipv4/ip_forward
$ cat /proc/sys/net/ipv4/ip_forward
1
==============================================================
# ping 통신 테스트
# 터미널1 (RED 11.11.11.2) >> ping 왜 실패했을까요?
$ ping -c 1 11.11.11.3
PING 11.11.11.3 (11.11.11.3) 56(84) bytes of data.
--- 11.11.11.3 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
# 터미널2 (호스트)
$ tcpdump -l -i br0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:11:15.990975 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 7769, seq 1, length 64
# 4개의 패킷이 DROP되었다고 표기되고 있음
# 패킷 전달 확인 (호스트)
$ watch -d 'iptables -v --numeric --table filter --list FORWARD'
Every 2.0s: iptables -v --numeric --table filter --list FORWARD docker1: Fri Jan 14 11:12:27 2022
Chain FORWARD (policy DROP 4 packets, 336 bytes)
pkts bytes target prot opt in out source destination
4 336 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
4 336 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
$ watch -d 'iptables -v --numeric --table filter --list FORWARD;echo;iptables -v --numeric --table filter --list DOCKER-USER;echo;iptables -v --numeric --table filter --list DOCKER-ISOLATION-STAGE-1'
Every 2.0s: iptables -v --numeric --table filter --list FORWARD;echo;iptables -v -... docker1: Fri Jan 14 11:13:05 2022
Chain FORWARD (policy DROP 4 packets, 336 bytes)
pkts bytes target prot opt in out source destination
4 336 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
4 336 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
4 336 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
4 336 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
# 터미널3 (BLUE 11.11.11.3)
$ tcpdump -l -i beth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on beth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:15:53.310290 ARP, Request who-has 11.11.11.3 tell 11.11.11.2, length 28
11:15:53.310414 ARP, Reply 11.11.11.3 is-at 9a:a3:cd:aa:1e:ea (oui Unknown), length 28
현재는 11.11.11.3 과 11.11.11.2간의 통신이 안되고 있습니다.
통신 실패 원인 분석
- Netfileter : 커널 모듈로 네트워크 패킷을 처리하는 프레임워크
- 네트워크 연산을 핸들러 형태로 처리할 수 있도록
hook 지점을 제공 - 패킷이 어떻게 전송될 지에 대한 결정 방법을 제공
- 네트워크 연산을 핸들러 형태로 처리할 수 있도록
- INPUT, OUTPUT
- FORWARD : 호스트 입장에서 ‘네트워크 네임스페이스’도 지나가는 패킷 처리(FORWARD)를 한다
Netfilter 과 IPtables
패킷은 netfliter(일종의 백엔드) 에서 파놓은 여러 hook 를 통과하는데 hook 별로 iptables 에서 정의한 각 체인룰을 점검합니다.
- iptables(일종의 프런트엔드 혹은 애플리케이션) : Netfileter 가 파놓은 hook 에 룰을 등록하고 관리하는 방법을 제공하는 툴
왜 Netfilter 과 iptables 처럼 구분해 놓을까요? 실제 iptables은 방화벽의 역할처럼 패킷을 차단, 허용하는 등의 필터링 기능을 수행하는 것은 아니며 리눅스 커널에 내장된 netfiler 라는 리눅스 커널 모듈을 통해 실제로 필터링이 동작합니다. iptables 는 netfilter 를 이용할 수 있도록 해주는 사용자 공간 응용 프로그램(User-Spac Application) 입니다.
- hook 에 정의된 체인은 테이블 순서대로 등록된 룰을 체크하고 조건을 만족하면 action(target)을 트리거합니다.
- table : 목적/용도별 rules 모음 → 예) filter, nat, mangle …
- chain : 패킷이 지나가는 hook 별로 존재 → 예) PREROUTING, FORWARD, INPUT … (넷필터의 hook 에 매핑됩니다)
- rule : table 과 chain matrix 에 대해서 정의 → 예) protocol type, dest/src.address, headers …
- action(target) : 패킷이 룰에 매칭되면 트리거됨 → 예) ACCEPT, DROP, REJECT … * -j : jump
RED → BLUE ping 허용 설정
- 호스트 입장에서는 “외부(RED, src) → 외부(BLUE, dst)” 패킷이므로 FORWARD 체인의 filter 테이블 룰에 허용 정책을 추가 해야 합니다.
# 터미널2 (호스트)
# iptables 설정 정보 확인
$ iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
$ iptables -t nat -S | grep '\-P'
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
# iptables 설정 추가 -t(table), -I(insert chain), -j(jump to - ACCEPT 허용)
$ iptables -t filter -I DOCKER-USER -j ACCEPT
$ iptables -nvL -t filter
Chain INPUT (policy ACCEPT 14 packets, 884 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
7 588 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
7 588 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 8 packets, 536 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
7 588 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
7 588 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
$ iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j ACCEPT
-A DOCKER-USER -j RETURN
$ iptables -t nat -S | grep '\-P'
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
==============================================================
# ping 통신 테스트
# 터미널1 (RED 11.11.11.2)
$ ping -c 1 11.11.11.3
PING 11.11.11.3 (11.11.11.3) 56(84) bytes of data.
64 bytes from 11.11.11.3: icmp_seq=1 ttl=64 time=0.121 ms
--- 11.11.11.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.121/0.121/0.121/0.000 ms
$ ip -c neigh
11.11.11.3 dev reth0 lladdr 9a:a3:cd:aa:1e:ea REACHABLE
# 터미널2 (호스트)
$ watch -d 'iptables -v --numeric --table filter --list FORWARD;echo;iptables -v --numeric --table filter --list DOCKER-USER;echo;iptables -v --numeric --table filter --list DOCKER-ISOLATION-STAGE-1'
Every 2.0s: iptables -v --numeric --table filter --list FORWARD;echo;iptables -v -... docker1: Fri Jan 14 12:03:31 2022
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
9 756 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
7 588 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
2 168 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
7 588 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
7 588 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
$ tcpdump -l -i br0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:03:58.838303 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 16010, seq 1, length 64
12:03:58.838356 IP 11.11.11.3 > 11.11.11.2: ICMP echo reply, id 16010, seq 1, length 64
## 정보 확인
$ ip -c neigh
10.0.2.2 dev enp0s3 lladdr 52:54:00:12:35:02 DELAY
10.0.2.3 dev enp0s3 lladdr 52:54:00:12:35:03 STALE
# 터미널3 (BLUE 11.11.11.3)
$ tcpdump -l -i beth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on beth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:05:06.573888 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 16065, seq 1, length 64
12:05:06.573909 IP 11.11.11.3 > 11.11.11.2: ICMP echo reply, id 16065, seq 1, length 64
$ ip -c neigh
11.11.11.2 dev beth0 lladdr 66:f9:63:30:ea:4a REACHABLE
==============================================================
# (옵션) 추가 방안1 : 출발지 IP 11.2, 11.3 허용
## -t(table), -A(APPEND chain rule), -s(출발지), -j(jump to - ACCEPT 허용)
$ iptables -t filter -A FORWARD -s 11.11.11.2/32 -j ACCEPT
$ iptables -t filter -A FORWARD -s 11.11.11.3/32 -j ACCEPT
## 추가 정책 삭제 시
$ iptables -t filter -D FORWARD -s 11.11.11.2/32 -j ACCEPT
$ iptables -t filter -D FORWARD -s 11.11.11.3/32 -j ACCEPT
# (옵션) 추가 방안2 : 11.11.11.0/24 대역 출발지 허용
$ iptables -t filter -A FORWARD -s 11.11.11.0/24 -j ACCEPT
## 추가 정책 삭제 시
$ iptables -t filter -D FORWARD -s 11.11.11.0/24 -j ACCEPT
# (옵션) 추가 방안3 : FORWARD 기본 정책 허용으로 변경
$ iptables -t filter -P FORWARD ACCEPT
## 추가 정책 삭제 시
$ iptables -t filter -P FORWARD DROP
RED/BLUE ==> 호스트 & 외부(인터넷) 통신
위의 실습을 진행하였을 경우 아래 명령어를 처리 하지 않아도 됩니다.
$ ip netns add RED
$ ip link add reth0 netns RED type veth peer name reth1
$ ip netns add BLUE
$ ip link add beth0 netns BLUE type veth peer name beth1
$ ip link add br0 type bridge
$ ip link set reth1 master br0
$ ip link set beth1 master br0
$ ip netns exec RED ip addr add 11.11.11.2/24 dev reth0
$ ip netns exec BLUE ip addr add 11.11.11.3/24 dev beth0
$ ip netns exec RED ip link set reth0 up; ip link set reth1 up;
$ ip netns exec BLUE ip link set beth0 up; ip link set beth1 up;
$ ip link set br0 up
# iptables -t filter -A FORWARD -s 11.11.11.0/24 -j ACCEPT
$ iptables -t filter -I DOCKER-USER -j ACCEPT
호스트에서 RED 나 BLUE 로 ping 통신 ==> RED 에서 외부로 통신
# 터미널1 (RED 11.11.11.2)
$ nsenter --net=/var/run/netns/RED
$ tcpdump -i any
# 터미널3 (호스트)
$ tcpdump -i br0 -n
# 터미널2 (호스트) >> 호스트에서 RED 로 통신이 안되는 이유가 무엇일까요?
$ ping -c 1 11.11.11.2
PING 11.11.11.2 (11.11.11.2) 56(84) bytes of data.
--- 11.11.11.2 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ ip -c route
default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.50.0/24 dev enp0s8 proto kernel scope link src 192.168.50.10
$ ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:31:14:45:32:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 67865sec preferred_lft 67865sec
inet6 fe80::31:14ff:fe45:3203/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:40:48:37 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.10/24 brd 192.168.50.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe40:4837/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:cd:05:9d:50 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
5: reth1@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 0e:7f:1d:23:77:01 brd ff:ff:ff:ff:ff:ff link-netns RED
inet6 fe80::c7f:1dff:fe23:7701/64 scope link
valid_lft forever preferred_lft forever
7: beth1@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 36:88:d0:d2:f6:03 brd ff:ff:ff:ff:ff:ff link-netns BLUE
inet6 fe80::3488:d0ff:fed2:f603/64 scope link
valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0e:7f:1d:23:77:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::c7f:1dff:fe23:7701/64 scope link
valid_lft forever preferred_lft forever
==============================================================
# 터미널2 (호스트) >> br0 에 IP 추가(라우팅 정보)
$ ip addr add 11.11.11.1/24 dev br0
$ ping -c 1 11.11.11.2
PING 11.11.11.2 (11.11.11.2) 56(84) bytes of data.
64 bytes from 11.11.11.2: icmp_seq=1 ttl=64 time=0.108 ms
--- 11.11.11.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.108/0.108/0.108/0.000 ms
$ ping -c 1 11.11.11.3
PING 11.11.11.3 (11.11.11.3) 56(84) bytes of data.
64 bytes from 11.11.11.3: icmp_seq=1 ttl=64 time=0.113 ms
--- 11.11.11.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.113/0.113/0.113/0.000 ms
# 터미널1 (RED 11.11.11.2) >> 10.0.2.15 와 통신이 안되는 이유는 무엇일까요?
$ ping -c 1 11.11.11.1
PING 11.11.11.1 (11.11.11.1) 56(84) bytes of data.
64 bytes from 11.11.11.1: icmp_seq=1 ttl=64 time=0.050 ms
--- 11.11.11.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.050/0.050/0.050/0.000 ms
$ ping -c 1 10.0.2.15
ping: connect: Network is unreachable
$ ip -c route
11.11.11.0/24 dev reth0 proto kernel scope link src 11.11.11.2
$ ip -c addr
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
6: reth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 66:f9:63:30:ea:4a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 11.11.11.2/24 scope global reth0
valid_lft forever preferred_lft forever
inet6 fe80::64f9:63ff:fe30:ea4a/64 scope link
valid_lft forever preferred_lft forever
==============================================================
# 터미널3 (호스트)
$ tcpdump -i any icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:22:53.664815 IP 11.11.11.2 > 10.0.2.15: ICMP echo request, id 16949, seq 1, length 64
12:22:53.664815 IP 11.11.11.2 > 10.0.2.15: ICMP echo request, id 16949, seq 1, length 64
12:22:53.665046 IP 10.0.2.15 > 11.11.11.2: ICMP echo reply, id 16949, seq 1, length 64
12:22:53.665049 IP 10.0.2.15 > 11.11.11.2: ICMP echo reply, id 16949, seq 1, length 64
# 터미널1 (RED 11.11.11.2)
$ ip route add default via 11.11.11.1
$ ping -c 1 10.0.2.15
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.248 ms
--- 10.0.2.15 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.248/0.248/0.248/0.000 ms
$ ip -c route
default via 11.11.11.1 dev reth0
11.11.11.0/24 dev reth0 proto kernel scope link src 11.11.11.2
# 외부 대역(인터넷)과 통신이 안되는 이유가 무엇일까요?
$ ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
# 터미널2 (호스트)
$ iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
$ iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 19 packets, 1516 bytes)
pkts bytes target prot opt in out source destination
8 592 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 8 packets, 592 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 31 packets, 2355 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 35 packets, 2691 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
## POSTROUTING : 라우팅 Outbound or 포워딩 트래픽에 의해 트리거되는 netfilter hook
## POSTROUTING 에서는 SNAT(Source NAT) 설정
$ iptables -t nat -A POSTROUTING -s 11.11.11.0/24 -j MASQUERADE
$ watch -d 'iptables -v --numeric --table nat --list POSTROUTING'
Every 2.0s: iptables -v --numeric --table nat --list POSTROUTING docker1: Fri Jan 14 12:26:17 2022
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE all -- * * 11.11.11.0/24 0.0.0.0/0
$ iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 19 packets, 1516 bytes)
pkts bytes target prot opt in out source destination
8 592 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 8 packets, 592 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 31 packets, 2355 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 35 packets, 2691 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
root@docker1:~# iptables -t nat -A POSTROUTING -s 11.11.11.0/24 -j MASQUERADE
root@docker1:~# watch -d 'iptables -v --numeric --table nat --list POSTROUTING'
root@docker1:~# iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
8 592 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE all -- * * 11.11.11.0/24 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
$ conntrack -L --src-nat
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been shown.
# 터미널1 (RED 11.11.11.2)
$ ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=39.6 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 39.563/39.563/39.563/0.000 ms
# 터미널3 (BLUE 11.11.11.3)
$ nsenter --net=/var/run/netns/BLUE
$ ip route add default via 11.11.11.1
$ ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=37.1 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 37.113/37.113/37.113/0.000 ms
# 삭제
$ ip netns delete RED
$ ip netns delete BLUE
$ ip link delete br0
$ iptables -t filter -D DOCKER-USER -j ACCEPT
$ iptables -t nat -D POSTROUTING -s 11.11.11.0/24 -j MASQUERADE
네트워크 네임스페이스만으로도 직접 수동으로 설정할 경우 복잡합니다. 도커는 컨테이너 실행(run) 시 자동으로 해당 컨테이너의 네임스페이스(네트워크, 마운트, PID 등)를 생성(삭제 등)하여 호스트와 격리해줍니다.
도커 네트워크 모델
- 기본 네트워크 모드 :
Bridge,Host, None + 추가 네트워크 플러그인 : macvlan, ipvlan, overlay
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6988f789afeb bridge bridge local
933453f3242a host host local
e3361781c03e none null local
$ docker info | grep Network
Network: bridge host ipvlan macvlan null overlay
- 컨테이너 기본 생성 시 자동으로 docker0 bridge를 사용
- 기본 172.17.0.0/16 대역을 컨테이너가 사용, 대역 변경 설정 가능
- Host 는 호스트의 환경을 그대로 사용. 애플리케이션 별도 포트 포워딩 없이 바로 서비스 가능. 컨테이너의 호스트 이름도 호스트 머신의 이름과 동일
- None 는 말 그래도 아무런 네트워크를 쓰지 않는 것. 외부와의 연결이 단절됨. 컨테이너 내부에 lo 인터페이스만 존재
도커 호스트가 다수 일 때, 컨테이너들 끼리 직접 통신을 하기 위해서는 네트워크 대역이 중복되지 않게 설정해야 되고 overlay 혹은 직접 라우팅이 가능하게 설정 등이 필요합니다. 이런 부분을 쿠버네티스에서는 CNI(Network Interface) 플러그인 (예. calico 등)이 처리하게 됩니다.
끝으로
도커 네트워크에서 macvlan, ipvlan, overlay의 설명은 참고 링크를 확인하여 주세요.
위의 예제는 대부분 KANG에서 실습한 예제 입니다.
1주에 생각보다 많이 배워서 소화하는데 시간이 오래 걸렷네요. 2주차 스터디가 기대됩니다.