оно 32 символа сейчас

надо больше

65536

login:login-000 FATAL Exited too quickly (process log may have details) omap FATAL Exited too quickly (process log may have details) pmwriter:pmwriter-000 FATAL Exited too quickly (process log may have details) pmwriter:pmwriter-001 FATAL Exited too quickly (process log may have details) sae:sae-000 FATAL Exited too quickly (process log may have details) sae:sae-001 FATAL Exited too quickly (process log may have details) scheduler FATAL Exited too quickly (process log may have details)

обновился

@freeseacher Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp:

IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c' Traceback (most recent call last): File "./services/web/service.py", line 17, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c'

Проверь дира создана не?

В последнем патче оно есть

cffi нет

/opt/noc/lib/python2.7/site-packages/cffi - тут есть

Проверь дира создана не?

/opt/noc/lib/python2.7/site-packages/__pycache__/ нету

Noc-activator стал зависать.

Молча перестаёт в лог писать и все. Пока не отстрелишь -9 и заново но не запустишь так и висит.

Часа на два хватает и снова зависает.

Часа на два хватает и снова зависает.

Проверь cpu usage

Проверь в нетстат close_wait

Думаю в close_wait активаторы у тя

Всплеск это я его перезапустил

Close_wait думаю было, но после первого раза сделал

net.core.somaxconn=5000 net.ipv4.ip_local_port_range=15000 61000 net.ipv4.tcp_fin_timeout=30 net.ipv4.tcp_tw_recycle=1 net.ipv4.tcp_tw_reuse=1 net.core.netdev_max_backlog=2000 net.ipv4.tcp_max_syn_backlog=2048

Я в другом месте наступал на эти грабли. Есть такая хренька Air Control 2 для Ubiquiti

Количество открытых файлов на всякий 4096

Система Centos 7

Noc естно там один в виртуалке живёт. Правда у нас vmware перегружено. Достаточно резиновое все.

Close_wait думаю было, но после первого раза сделал

Это не должно помочь

Локалрендж плохой

Нок сам юзает 19к порты

С месяц я эти sysctl выставляю в правильные

Правильные для Нок это примерно что?

Локалрендж уберу.

Сейчас он бодро шуршит и Close_wait нет

Хотя в логе активатора навалом Timeout

Сейчас картина такая.

Локалрендж уберу.

Можно не убирать просто не юзаются они

А картинка с чего?

Какой-то ррд

Zenoss

Ага

В своп не лезет, 4 гига я смотрю достаточно.

Надо понять на чем висит

Ща поищу команды

Постгри и монгу не крутил. По умолчанию все.

/dev/mapper/DB-NOCDB 50G 9.5G 38G 21% /opt/DB

Правда selinux включен

root@wrk01_moscow noc]# ./noc rpc --hint=10.36.129.11:19000 ctl.prof_start Profiling started [root@wrk01_moscow noc]# ./noc rpc --hint=10.36.129.11:19000 ctl.prof_funcs Clock type: CPU Ordered by: totaltime, desc name ncall tsub ttot tavg ../noc/lib/python2.7/site-packages/concurrent/futures/thread.py:50 _WorkItem.run 112/109 0.005184 2.550023 0.022768 /opt/noc/lib/python2.7/site-packages/tornado/stack_context.py:271 null_wrapper 3080/3078 0.028438 2.542081 0.000825 /opt/noc/services/activator/api/activator.py:26 ActivatorAPI.script 112/109 0.006002 2.518217 0.022484 /opt/noc/core/script/base.py:166 Script.run 113/109 0.009225 2.450862 0.021689 ..oc/lib/python2.7/site-packages/tornado/ioloop.py:594 EPollIOLoop._run_callback 2001/2000 0.010707 1.964195 0.000982 /opt/noc/lib/python2.7/site-packages/tornado/ioloop.py:400 EPollIOLoop.run_sync 253/250 0.006090 1.878874 0.007426 ... [root@wrk01_moscow noc]# ./noc rpc --hint=10.36.129.11:19000 ctl.prof_stop Profiling stopped

Я нарисовал правила

Натравливай это на бажный активатор

Появится файл с люком

В tmp

Manhole

На него иди socat

По доке от manhole

Видимо я не все понимаю

[root@noc etc]# cd /opt/noc [root@noc noc]# ./noc rpc Traceback (most recent call last): File "manage.py", line 35, in <module> import noc.urls File "/opt/noc/urls.py", line 14, in <module> init_solutions() File "/opt/noc/lib/solutions.py", line 53, in init_solutions CustomField.install_fields() File "/opt/noc/main/models/customfield.py", line 283, in install_fields for f in cls.objects.filter(is_active=True).order_by("table"): File "/opt/noc/lib/python2.7/site-packages/django/db/models/query.py", line 118, in _result_iter self._fill_cache() File "/opt/noc/lib/python2.7/site-packages/django/db/models/query.py", line 892, in _fill_cache self._result_cache.append(self._iter.next()) File "/opt/noc/lib/python2.7/site-packages/django/db/models/query.py", line 291, in iterator for row in compiler.results_iter(): File "/opt/noc/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 763, in results_iter for rows in self.execute_sql(MULTI): File "/opt/noc/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 817, in execute_sql cursor = self.connection.cursor() File "/opt/noc/lib/python2.7/site-packages/django/db/backends/__init__.py", line 319, in cursor cursor = util.CursorWrapper(self._cursor(), self) File "/opt/noc/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor self.connection = Database.connect(**conn_params) File "/opt/noc/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect conn = _connect(dsn, connection_factory=connection_factory, async=async) psycopg2.OperationalError: FATAL: Peer authentication failed for user "noc"

Мне для непонятливых пожалуйста :)

Что значит натравливай?

Понимаю что это питоновая терминология.

Алексей Широких ? :)

Увы пока подробнее не могу

В дороге

Если будет возможность. Пожалуйста.

Traceback (most recent call last): File "./services/sae/service.py", line 13, in <module> from noc.core.service.base import Service File "/opt/noc/core/service/base.py", line 32, in <module> from atomiclong import AtomicLong File "/opt/noc/lib/python2.7/site-packages/atomiclong.py", line 25, in <module> """) File "/opt/noc/lib/python2.7/site-packages/cffi/api.py", line 437, in verify lib = self.verifier.load_library() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 112, in load_library self._write_source() File "/opt/noc/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _write_source with open(self.sourcefilename, "w") as fp: IOError: [Errno 2] No such file or directory: '/opt/noc/lib/python2.7/site-packages/__pycache__/_cffi__x71c76ad5xf96f5519.c'

ls /opt/noc/lib/python2.7/site-packages/__pycache__/ ls: cannot access /opt/noc/lib/python2.7/site-packages/__pycache__/: No such file or directory

Опять завис активатор

%Cpu(s): 5.0 us, 0.0 sy, 0.0 ni, 95.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

Один CLOSE_WAIT висит. Ssh к юбиквити кстати

В этот раз на час хватило

Total DISK READ : 0.00 B/s | Total DISK WRITE : 34.36 K/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 49.63 K/s

Активатор завис на девайсе с плохим коннектом.

народ, кто джуны как брас использует ?

Мы используем - 2 штуки, тридцать с чем-то тысяч клиентов, IPoE

на весрсию 15.1R4 обновлялся ?

на весрсию 15.1R4 обновлялся ?

Рано для продуктива

нет еще

пока страшно

14.2R1.9

у меня выбора особого нету на 14.2R2 у него майка загибается

но в 15.1 говорят все очень круто

майка?

Mem: 1280M Active, 78M Inact, 192M Wired, 408M Cache, 112M Buf, 25M Free Swap: 2837M Total, 2837M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 29877 mitya 1 20 0 4644K 2976K pause 0:00 1.03% csh 1604 root 11 96 0 20920K 11160K ucond 179.7H 0.98% clksyncd 1667 root 1 96 0 102M 56432K select 103.4H 0.98% jdhcpd 1668 root 1 96 0 237M 191M select 114.9H 0.05% authd 1599 root 1 96 0 51272K 43720K RUN 59.7H 0.05% ppmd

сколько клиентов? и какой РЕ?

около 5-8K клиентов что такое PE ?

раутинг енджин. у нас RE-S-1800x4

а вообще какой джун?

х.з. у меня MX5-T

ааа

на МХ-80 вообще ограничение 8К макс

в лучшем случае

это маркетологи придумали

а DHCP relay или сервер на нем?

mitya# run show services service-sets summary Service sets CPU Interface configured Bytes used Policy bytes used utilization ms-0/2/0 2 1906603676 (15.78%) 4119352 ( 0.76%) 15.68 %

на МХ-80 вообще ограничение 8К макс

до 15-й ветки. И смотря как юзать. Если влан на юзера - да, где-то так. Если шаред - больше. Там все на ifl завязано в mpc, а не только на оперативку в RE. C 15.1 все переделали, теперь там все на flow-сущности и супердемон

dhcp сервер

ага

тоесть ты уже перешел?