будут рады почте)

@dvolodin события в fmwriter только ростут

очередь ростёт

мы все умрем аааааа

nsq поставь побольше

у меня гдето штуп 5 или 6

nsq? серьезно?

а почему нет

а покажи свой

# ./noc shell Python 2.7.5 (default, Sep 15 2016, 22:37:39) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> print (sys.path)

ноя 10 22:13:22 noc nsqd[844]: [nsqd] 2016/11/10 22:13:22.627259 CLIENT(127.0.0.1:46594): desired protocol magic ' V2'

MAGIC

Очереди ростут только

дело не в nsq, а в классификаторах

они неуспевают очередь разобрать почему-то

У меня 140 процессов классификаторов

это бред на мой взгляд

оставь его минут на 15 в покое. и следи за лате таск.

пусть разгребается

late нету

вроде норм.

после вечернего обновления очереди только ростут

После рестарта сервисов очереди идут на спад, но быстро наполняются

значит чего то не хватает.

может потоков активаторов или дискавери

до этого решалось кол-ом классификаторов

может потоков активаторов или дискавери

у него не те очереди, не задач, а событий

да

короче нужен тест чистого деплоя за проксёй

тут я пока пас. хз. если только кто-нибудь покажет как в докере 1 строчкой прокси запустить =)

ага

кто-то классификатор порвал

@dvolodin 2016-11-10 22:48:26,293 [nsq.reader] [noc:4150:events:fmwriter] uncaught exception while handling message 0711f11dbe435015 body:'{"object":120,"data":{"source":"syslog","message":"Nov 10 2016 22:06:38: %ASA-2-106006: Deny inbound UDP from 10.19.127.7\\/137 to 10.80.16.220\\/137 on interface inside","collector":"default"},"ts":1478804883}' Traceback (most recent call last): File "/opt/noc/lib/python2.7/site-packages/nsq/reader.py", line 344, in _handle_message success = self.process_message(message) File "/opt/noc/lib/python2.7/site-packages/nsq/reader.py", line 707, in process_message return self.message_handler(message) File "/opt/noc/core/service/base.py", line 488, in call_json_handler r = handler(message, **data) File "./services/classifier/service.py", line 754, in on_event mo = ManagedObject.get_by_id(object) File "/opt/noc/core/cache/decorator.py", line 73, in wrapper x_cache.set(k, v, ttl=ttl) File "/opt/noc/core/cache/mongo.py", line 59, in set expires = datetime.datetime.now() + datetime.timedelta(seconds=ttl) TypeError: unsupported type for timedelta seconds component: NoneType

2016-11-10 22:49:39,110 [nsq.reader] [noc:4150:events:fmwriter] uncaught exception while handling message 0711f3c07e835009 body:'{"object":119,"data":{"source":"syslog","message":"1420: Nov 10 22:21:40: %LINK-3-UPDOWN: Interface GigabitEthernet1\\/0\\/17, changed state to up","collector":"default"},"ts":1478805701}' Traceback (most recent call last): File "/opt/noc/lib/python2.7/site-packages/nsq/reader.py", line 344, in _handle_message success = self.process_message(message) File "/opt/noc/lib/python2.7/site-packages/nsq/reader.py", line 707, in process_message return self.message_handler(message) File "/opt/noc/core/service/base.py", line 488, in call_json_handler r = handler(message, **data) File "./services/classifier/service.py", line 754, in on_event mo = ManagedObject.get_by_id(object) File "/opt/noc/core/cache/decorator.py", line 73, in wrapper x_cache.set(k, v, ttl=ttl) File "/opt/noc/core/cache/mongo.py", line 59, in set expires = datetime.datetime.now() + datetime.timedelta(seconds=ttl) TypeError: unsupported type for timedelta seconds component: NoneType

2016-11-10 22:54:18,724 [nsq.reader] [noc:4150:events:fmwriter] uncaught exception while handling message 0711ee9455835214 body:'{"object":22,"data":{"source":"system","probe":"ping","result":"success","ip":"10.100.5.101"},"ts":1478804754.9401609898}' Traceback (most recent call last): File "/opt/noc/lib/python2.7/site-packages/nsq/reader.py", line 344, in _handle_message success = self.process_message(message) File "/opt/noc/lib/python2.7/site-packages/nsq/reader.py", line 707, in process_message return self.message_handler(message) File "/opt/noc/core/service/base.py", line 488, in call_json_handler r = handler(message, **data) File "./services/classifier/service.py", line 754, in on_event mo = ManagedObject.get_by_id(object) File "/opt/noc/core/cache/decorator.py", line 73, in wrapper x_cache.set(k, v, ttl=ttl) File "/opt/noc/core/cache/mongo.py", line 59, in set expires = datetime.datetime.now() + datetime.timedelta(seconds=ttl) TypeError: unsupported type for timedelta seconds component: NoneType 2016-11-10 22:54:18,725 [nsq.reader] [events:fmwriter] backing off for 36.61 seconds (1 connections)

@dvolodin это поважнее ip sla будет

Dmitry Volodin pushed to noc/feature/microservices correlator: Fix process name

поправил

Dmitry Volodin pushed to noc/feature/microservices cache: Set default ttl for mongo backend

и process name и default ttl для монги

для memcached еще нужны будут роли

с пакетами под разные системы

пролечилось?

@dvolodin обрадовал, спасибо!

draft-keyupate-idr-bgp-spf-01 - Shortest Path Routing Extensions for BGP Protocol https://tools.ietf.org/html/draft-keyupate-idr-bgp-spf-01

Dmitry Volodin pushed to noc/feature/microservices card: heatmap pop links optimization

Dmitry Volodin pushed to noc/feature/microservices Cache active events when passing from classifier to correlator

:-(

Starting job #263 ERROR! Attempted to execute "/opt/tower/bin/tower-inv" as inventory script: Inventory script (/opt/tower/bin/tower-inv) had an execution error: Traceback (most recent call last): File "/opt/tower/bin/tower-inv", line 11, in <module> load_entry_point('noc-tower==0.1a61', 'console_scripts', 'tower-inv')() File "/opt/tower/lib/python2.7/site-packages/tower/cli/inv.py", line 38, in main ansible_list(options, args) File "/opt/tower/lib/python2.7/site-packages/tower/cli/inv.py", line 61, in ansible_list print json.dumps(env.ansible_inventory()) File "/opt/tower/lib/python2.7/site-packages/tower/models/environment.py", line 246, in ansible_inventory node_services[s.node.name] += [s] File "/opt/tower/lib/python2.7/site-packages/peewee.py", line 1205, in __get__ return self.get_object_or_id(instance) File "/opt/tower/lib/python2.7/site-packages/peewee.py", line 1196, in get_object_or_id obj = self.rel_model.get(self.field.to_field == rel_id) File "/opt/tower/lib/python2.7/site-packages/peewee.py", line 4505, in get return sq.get() File "/opt/tower/lib/python2.7/site-packages/peewee.py", line 2889, in get % self.sql()) tower.models.node.NodeDoesNotExist: Instance matching query does not exist: SQL: SELECT "t1"."id", "t1"."environment_id", "t1"."datacenter_id", "t1"."node_type_id", "t1"."name", "t1"."description", "t1"."address", "t1"."login_as", "t1"."is_enabled" FROM "node" AS t1 WHERE ("t1"."id" = ?) PARAMS: [3]

удаляешь нодо в башне и при деплое валится.

создаёшь ноду а у неё уже настройки которые были раньше .

yum install libssh2-devel

[root@jjjj noc]# ./scripts/deploy/compile-bytecode Sorry: IndentationError: unexpected indent (get_interface_status.py, line 47) CHANGED

типа отчета в fm есть что-либо?

не удаляй ноды. башня разваливается. лучше сервисы все ноды убирай

пустой список для ping при пустой базе ➔

типа отчета в fm есть что-либо?

Да, были

чет не найду. есть такой -Сводка событий: По классу события. а какая железка событие вызвала - нету

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in teroni/noc-2/teroni/-ping--1478844040376 (pull request #268) пустой список для ping_sources, trap_sources, syslog_sources при пустой базе teroni: пустой список для ping_sources, trap_sources, syslog_sources при пустой базе teroni: пустой список для ping при пустой базе

пустой список для ping при пустой базе ➔

✅ Merged by Dmitry Volodin

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in teroni/noc-2/teroni/-ping--1478844040376 (pull request #268) пустой список для ping_sources, trap_sources, syslog_sources при пустой базе teroni: пустой список для ping_sources, trap_sources, syslog_sources при пустой базе teroni: пустой список для ping при пустой базе

teroni pushed to noc/teroni/-ping--1478844040376 teroni: пустой список для ping_sources, trap_sources, syslog_sources при пустой базе teroni: пустой список для ping при пустой базе Dmitry Volodin: Cache active events when passing from classifier to correlator Dmitry Volodin: card: heatmap pop links optimization Dmitry Volodin: cache: Set default ttl for mongo backend ... See all

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in pnpwin/noc-microservices-test/collections/fm.eventclassificationrules/Juniper/JUNOS/ (pull request #263) Rename and add some fm.event.classrules for Junos Ilya Somov: fix conflict "entering config mode" and "commit complete" Ilya Somov: Fix Config changed and add #2 Ilya Somov: Rename and add some fm.event.classrules for Junos

Rename and add some fm.event.classrules for Junos ➔

✅ Merged by Dmitry Volodin

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in pnpwin/noc-microservices-test/collections/fm.eventclassificationrules/Juniper/JUNOS/ (pull request #263) Rename and add some fm.event.classrules for Junos Ilya Somov: fix conflict "entering config mode" and "commit complete" Ilya Somov: Fix Config changed and add #2 Ilya Somov: Rename and add some fm.event.classrules for Junos

Ilya Somov pushed to noc/collections/fm.eventclassificationrules/Juniper/JUNOS/ Ilya Somov: fix conflict "entering config mode" and "commit complete" Ilya Somov: Fix Config changed and add #2 Ilya Somov: Rename and add some fm.event.classrules for Junos Dmitry Volodin: AlarmSeverity.get_severity(): Speed optimization Dmitry Luhtionov: Support for AO and AYT command in telnet ... See all

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in aversant/noc2_cl/Scripts/AddLinux_RHEL (pull request #229) Add Linux.RHEL ported profiles Andrey: Fix get_cdp_neighbors Andrey: Add Linux.RHEL ported profiles

✅ Merged by Dmitry Volodin

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in aversant/noc2_cl/Scripts/AddLinux_RHEL (pull request #229) Add Linux.RHEL ported profiles Andrey: Fix get_cdp_neighbors Andrey: Add Linux.RHEL ported profiles

Andrey pushed to noc/Scripts/AddLinux_RHEL Andrey: Fix get_cdp_neighbors Andrey: Add Linux.RHEL ported profiles Dmitry Volodin: discovery: Fix interface profile metrics cache Dmitry Volodin: Object paths for analytics queries Dmitry Volodin: discovery: Fix metrics collection ... See all

@dvolodin pmwriter-000.log крешится http://pastebin.com/niE7zuVS

Dmitry Volodin pushed to noc/feature/microservices Move cached credentials to the caching infrastructure

buffer size ➔

Dmitry Volodin pushed to noc/feature/microservices Merged in teroni/noc-2/feature/microservices (pull request #269) buffer size

buffer size ➔

✅ Merged by Dmitry Volodin

Dmitry Volodin pushed to noc/feature/microservices Dmitry Volodin: Merged in teroni/noc-2/feature/microservices (pull request #269) buffer size teroni: buffer size

,,

@dvolodin ты есть? memcached обязателен теперь?

https://bitbucket.org/nocproject/noc/src/abca3752f1e6fafa091bf61956b4149495262df5/sa/profiles/Maipu/?at=develop перетащите в микросервисы

нужны роли для memcached под RHEL и Debian

если нет мемкеша - положит все в монгу

Wad судя по всему нет

если нет мемкеша - будет кешировать в монгу

в принципе, он и сейчас это делает

не стартует [nsq.reader] [correlator.dispose.default:dispose] ошибка query error: HTTP 500: Internal Server Error

и так во все где пользуется nsq

это нормально для него

свежая инсталяция, последний апдейт

@dvolodin как мне полечить from noc.core.service.base import Service ImportError: No module named noc.core.service.base Traceback (most recent call last): File "./services/activator/service.py", line 11, in <module> from noc.core.service.base import Service ImportError: No module named noc.core.service.base

[root@nochost3 noc]# ./noc shell Python 2.7.5 (default, Sep 15 2016, 22:37:39) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. »> import sys »> print (sys.path) ['', '/opt/noc/lib64/python27.zip', '/opt/noc/lib64/python2.7', '/opt/noc/lib64/python2.7/plat-linux2', '/opt/noc/lib64/python2.7/lib-tk', '/opt/noc/lib64/python2.7/lib-old', '/opt/noc/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7', '/usr/lib/python2.7', '/opt/noc/lib/python2.7/site-packages'] »>

франкенштейна собираем?

noc.pth посмотри

где?

»> print (noc.pth) Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'noc' is not defined »>

lib/python2.7/site-packages/noc.pth

cat lib/python2.7/site-packages/noc.pth # NOC paths import os; os.environ['DJANGO_SETTINGS_MODULE'] = 'noc.settings' /Users/dv/Documents/Projects/noc/tip/noc /Users/dv/Documents/Projects/noc/tip

это у меня