2019-10-31 04:42:06,232 DEBUG Dhanu => PING(20) 2019-10-31 04:42:06,934 INFO scheduling health checks: 2019-10-31 04:42:09,313 INFO scheduling jobs: 2019-10-31 04:42:09,316 DEBUG - qemu 2019-10-31 04:42:11,809 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:42:11,810 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:42:13,226 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:42:22,993 INFO [INIT] Dropping privileges 2019-10-31 04:42:22,993 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:42:22,993 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:42:22,995 INFO [INIT] Marking all workers as offline 2019-10-31 04:42:23,028 INFO [INIT] LAVA master has started. 2019-10-31 04:42:23,028 INFO [INIT] Using protocol version 3 2019-10-31 04:42:26,702 DEBUG Dhanu => PING(20) 2019-10-31 04:42:26,703 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:42:32,652 DEBUG lava-logs => PING(20) 2019-10-31 04:42:43,039 INFO scheduling health checks: 2019-10-31 04:42:43,071 INFO scheduling jobs: 2019-10-31 04:42:43,073 DEBUG - qemu 2019-10-31 04:42:43,133 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:42:43,133 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:42:43,138 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:42:44,315 INFO [INIT] Dropping privileges 2019-10-31 04:42:44,315 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:42:44,315 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:42:44,317 INFO [INIT] Marking all workers as offline 2019-10-31 04:42:44,375 INFO [INIT] LAVA master has started. 2019-10-31 04:42:44,375 INFO [INIT] Using protocol version 3 2019-10-31 04:42:46,720 DEBUG Dhanu => PING(20) 2019-10-31 04:42:46,721 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:42:52,672 DEBUG lava-logs => PING(20) 2019-10-31 04:43:04,379 INFO scheduling health checks: 2019-10-31 04:43:04,407 INFO scheduling jobs: 2019-10-31 04:43:04,408 DEBUG - qemu 2019-10-31 04:43:04,481 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:43:04,481 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:43:04,486 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:43:05,825 INFO [INIT] Dropping privileges 2019-10-31 04:43:05,826 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:43:05,826 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:43:05,827 INFO [INIT] Marking all workers as offline 2019-10-31 04:43:05,853 INFO [INIT] LAVA master has started. 2019-10-31 04:43:05,853 INFO [INIT] Using protocol version 3 2019-10-31 04:43:06,737 DEBUG Dhanu => PING(20) 2019-10-31 04:43:06,738 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:43:12,688 DEBUG lava-logs => PING(20) 2019-10-31 04:43:25,867 INFO scheduling health checks: 2019-10-31 04:43:25,897 INFO scheduling jobs: 2019-10-31 04:43:25,899 DEBUG - qemu 2019-10-31 04:43:25,973 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:43:25,974 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:43:25,985 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:43:27,103 INFO [INIT] Dropping privileges 2019-10-31 04:43:27,103 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:43:27,103 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:43:27,105 INFO [INIT] Marking all workers as offline 2019-10-31 04:43:27,144 INFO [INIT] LAVA master has started. 2019-10-31 04:43:27,144 INFO [INIT] Using protocol version 3 2019-10-31 04:43:27,278 DEBUG Dhanu => PING(20) 2019-10-31 04:43:27,278 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:43:32,707 DEBUG lava-logs => PING(20) 2019-10-31 04:43:47,158 INFO scheduling health checks: 2019-10-31 04:43:47,200 INFO scheduling jobs: 2019-10-31 04:43:47,203 DEBUG - qemu 2019-10-31 04:43:47,282 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:43:47,282 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:43:47,292 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:43:48,534 INFO [INIT] Dropping privileges 2019-10-31 04:43:48,534 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:43:48,534 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:43:48,536 INFO [INIT] Marking all workers as offline 2019-10-31 04:43:48,568 INFO [INIT] LAVA master has started. 2019-10-31 04:43:48,568 INFO [INIT] Using protocol version 3 2019-10-31 04:43:48,670 DEBUG Dhanu => PING(20) 2019-10-31 04:43:48,671 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:43:52,723 DEBUG lava-logs => PING(20) 2019-10-31 04:44:08,585 INFO scheduling health checks: 2019-10-31 04:44:08,613 INFO scheduling jobs: 2019-10-31 04:44:08,615 DEBUG - qemu 2019-10-31 04:44:08,688 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:44:08,688 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:44:08,693 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:44:09,795 INFO [INIT] Dropping privileges 2019-10-31 04:44:09,795 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:44:09,795 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:44:09,797 INFO [INIT] Marking all workers as offline 2019-10-31 04:44:09,871 INFO [INIT] LAVA master has started. 2019-10-31 04:44:09,872 INFO [INIT] Using protocol version 3 2019-10-31 04:44:12,738 DEBUG lava-logs => PING(20) 2019-10-31 04:44:28,714 DEBUG Dhanu => PING(20) 2019-10-31 04:44:28,714 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:44:29,873 INFO scheduling health checks: 2019-10-31 04:44:29,892 INFO scheduling jobs: 2019-10-31 04:44:29,894 DEBUG - qemu 2019-10-31 04:44:29,964 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:44:29,964 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:44:29,969 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:44:31,027 INFO [INIT] Dropping privileges 2019-10-31 04:44:31,027 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:44:31,027 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:44:31,029 INFO [INIT] Marking all workers as offline 2019-10-31 04:44:31,062 INFO [INIT] LAVA master has started. 2019-10-31 04:44:31,062 INFO [INIT] Using protocol version 3 2019-10-31 04:44:32,752 DEBUG lava-logs => PING(20) 2019-10-31 04:44:48,736 DEBUG Dhanu => PING(20) 2019-10-31 04:44:48,737 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:44:51,064 INFO scheduling health checks: 2019-10-31 04:44:51,094 INFO scheduling jobs: 2019-10-31 04:44:51,096 DEBUG - qemu 2019-10-31 04:44:51,167 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:44:51,167 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:44:51,172 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:44:52,284 INFO [INIT] Dropping privileges 2019-10-31 04:44:52,284 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:44:52,284 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:44:52,285 INFO [INIT] Marking all workers as offline 2019-10-31 04:44:52,330 INFO [INIT] LAVA master has started. 2019-10-31 04:44:52,330 INFO [INIT] Using protocol version 3 2019-10-31 04:44:52,765 DEBUG lava-logs => PING(20) 2019-10-31 04:46:16,990 DEBUG Dhanu => PING(20) 2019-10-31 04:46:16,990 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:46:17,028 ERROR [STATE] lava-logs goes OFFLINE 2019-10-31 04:46:17,039 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 04:46:21,020 DEBUG lava-logs => PING(20) 2019-10-31 04:46:37,008 DEBUG Dhanu => PING(20) 2019-10-31 04:46:37,043 INFO scheduling health checks: 2019-10-31 04:46:37,063 INFO scheduling jobs: 2019-10-31 04:46:37,065 DEBUG - qemu 2019-10-31 04:46:37,142 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:46:37,142 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:46:37,150 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:46:38,574 INFO [INIT] Dropping privileges 2019-10-31 04:46:38,574 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:46:38,574 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:46:38,576 INFO [INIT] Marking all workers as offline 2019-10-31 04:46:38,610 INFO [INIT] LAVA master has started. 2019-10-31 04:46:38,611 INFO [INIT] Using protocol version 3 2019-10-31 04:46:41,035 DEBUG lava-logs => PING(20) 2019-10-31 04:46:57,030 DEBUG Dhanu => PING(20) 2019-10-31 04:46:57,031 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:46:58,612 INFO scheduling health checks: 2019-10-31 04:46:58,688 INFO scheduling jobs: 2019-10-31 04:46:58,691 DEBUG - qemu 2019-10-31 04:46:58,769 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:46:58,770 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:46:58,778 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:47:00,171 INFO [INIT] Dropping privileges 2019-10-31 04:47:00,171 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:47:00,171 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:47:00,173 INFO [INIT] Marking all workers as offline 2019-10-31 04:47:00,214 INFO [INIT] LAVA master has started. 2019-10-31 04:47:00,214 INFO [INIT] Using protocol version 3 2019-10-31 04:47:01,045 DEBUG lava-logs => PING(20) 2019-10-31 04:47:17,044 DEBUG Dhanu => PING(20) 2019-10-31 04:47:17,045 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:47:20,215 INFO scheduling health checks: 2019-10-31 04:47:20,353 INFO scheduling jobs: 2019-10-31 04:47:20,355 DEBUG - qemu 2019-10-31 04:47:20,461 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:47:20,462 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:47:20,471 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:47:22,017 INFO [INIT] Dropping privileges 2019-10-31 04:47:22,017 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:47:22,017 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:47:22,019 INFO [INIT] Marking all workers as offline 2019-10-31 04:47:22,091 INFO [INIT] LAVA master has started. 2019-10-31 04:47:22,091 INFO [INIT] Using protocol version 3 2019-10-31 04:47:22,102 DEBUG lava-logs => PING(20) 2019-10-31 04:47:37,066 DEBUG Dhanu => PING(20) 2019-10-31 04:47:37,066 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:47:42,097 INFO scheduling health checks: 2019-10-31 04:47:42,125 INFO scheduling jobs: 2019-10-31 04:47:42,127 DEBUG - qemu 2019-10-31 04:47:42,199 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:47:42,199 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:47:42,204 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:47:43,363 INFO [INIT] Dropping privileges 2019-10-31 04:47:43,364 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:47:43,364 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:47:43,365 INFO [INIT] Marking all workers as offline 2019-10-31 04:47:43,405 INFO [INIT] LAVA master has started. 2019-10-31 04:47:43,406 INFO [INIT] Using protocol version 3 2019-10-31 04:47:57,087 DEBUG Dhanu => PING(20) 2019-10-31 04:47:57,088 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:48:02,134 DEBUG lava-logs => PING(20) 2019-10-31 04:48:03,407 INFO scheduling health checks: 2019-10-31 04:48:03,471 INFO scheduling jobs: 2019-10-31 04:48:03,474 DEBUG - qemu 2019-10-31 04:48:03,559 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:48:03,559 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:48:03,564 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:48:04,942 INFO [INIT] Dropping privileges 2019-10-31 04:48:04,942 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:48:04,942 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:48:04,944 INFO [INIT] Marking all workers as offline 2019-10-31 04:48:04,961 INFO [INIT] LAVA master has started. 2019-10-31 04:48:04,962 INFO [INIT] Using protocol version 3 2019-10-31 04:48:17,100 DEBUG Dhanu => PING(20) 2019-10-31 04:48:17,101 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:48:22,148 DEBUG lava-logs => PING(20) 2019-10-31 04:48:24,965 INFO scheduling health checks: 2019-10-31 04:48:25,029 INFO scheduling jobs: 2019-10-31 04:48:25,032 DEBUG - qemu 2019-10-31 04:48:25,104 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:48:25,104 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:48:25,109 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:48:26,299 INFO [INIT] Dropping privileges 2019-10-31 04:48:26,300 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:48:26,300 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:48:26,301 INFO [INIT] Marking all workers as offline 2019-10-31 04:48:26,365 INFO [INIT] LAVA master has started. 2019-10-31 04:48:26,365 INFO [INIT] Using protocol version 3 2019-10-31 04:48:37,112 DEBUG Dhanu => PING(20) 2019-10-31 04:48:37,113 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:48:42,167 DEBUG lava-logs => PING(20) 2019-10-31 04:48:46,370 INFO scheduling health checks: 2019-10-31 04:48:46,397 INFO scheduling jobs: 2019-10-31 04:48:46,399 DEBUG - qemu 2019-10-31 04:48:46,472 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:48:46,472 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:48:46,477 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:48:47,536 INFO [INIT] Dropping privileges 2019-10-31 04:48:47,536 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:48:47,536 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:48:47,538 INFO [INIT] Marking all workers as offline 2019-10-31 04:48:47,601 INFO [INIT] LAVA master has started. 2019-10-31 04:48:47,601 INFO [INIT] Using protocol version 3 2019-10-31 04:48:57,130 DEBUG Dhanu => PING(20) 2019-10-31 04:48:57,130 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:49:02,184 DEBUG lava-logs => PING(20) 2019-10-31 04:49:07,607 INFO scheduling health checks: 2019-10-31 04:49:07,632 INFO scheduling jobs: 2019-10-31 04:49:07,634 DEBUG - qemu 2019-10-31 04:49:07,704 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:49:07,704 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:49:07,708 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:49:08,778 INFO [INIT] Dropping privileges 2019-10-31 04:49:08,779 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:49:08,779 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:49:08,780 INFO [INIT] Marking all workers as offline 2019-10-31 04:49:08,813 INFO [INIT] LAVA master has started. 2019-10-31 04:49:08,813 INFO [INIT] Using protocol version 3 2019-10-31 04:49:17,153 DEBUG Dhanu => PING(20) 2019-10-31 04:49:17,153 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:49:22,200 DEBUG lava-logs => PING(20) 2019-10-31 04:49:28,815 INFO scheduling health checks: 2019-10-31 04:49:28,857 INFO scheduling jobs: 2019-10-31 04:49:28,859 DEBUG - qemu 2019-10-31 04:49:28,934 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:49:28,935 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:49:28,939 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:49:30,041 INFO [INIT] Dropping privileges 2019-10-31 04:49:30,041 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:49:30,041 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:49:30,043 INFO [INIT] Marking all workers as offline 2019-10-31 04:49:30,070 INFO [INIT] LAVA master has started. 2019-10-31 04:49:30,071 INFO [INIT] Using protocol version 3 2019-10-31 04:49:37,172 DEBUG Dhanu => PING(20) 2019-10-31 04:49:37,173 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:49:42,212 DEBUG lava-logs => PING(20) 2019-10-31 04:49:50,080 INFO scheduling health checks: 2019-10-31 04:49:50,114 INFO scheduling jobs: 2019-10-31 04:49:50,116 DEBUG - qemu 2019-10-31 04:49:50,189 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:49:50,189 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:49:50,194 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:49:51,283 INFO [INIT] Dropping privileges 2019-10-31 04:49:51,284 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:49:51,284 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:49:51,285 INFO [INIT] Marking all workers as offline 2019-10-31 04:49:51,361 INFO [INIT] LAVA master has started. 2019-10-31 04:49:51,362 INFO [INIT] Using protocol version 3 2019-10-31 04:49:57,189 DEBUG Dhanu => PING(20) 2019-10-31 04:49:57,189 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:50:02,225 DEBUG lava-logs => PING(20) 2019-10-31 04:50:11,372 INFO scheduling health checks: 2019-10-31 04:50:11,405 INFO scheduling jobs: 2019-10-31 04:50:11,407 DEBUG - qemu 2019-10-31 04:50:11,479 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:50:11,479 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:50:11,484 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:50:12,786 INFO [INIT] Dropping privileges 2019-10-31 04:50:12,786 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:50:12,786 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:50:12,788 INFO [INIT] Marking all workers as offline 2019-10-31 04:50:12,830 INFO [INIT] LAVA master has started. 2019-10-31 04:50:12,830 INFO [INIT] Using protocol version 3 2019-10-31 04:50:17,209 DEBUG Dhanu => PING(20) 2019-10-31 04:50:17,210 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:50:22,244 DEBUG lava-logs => PING(20) 2019-10-31 04:50:32,839 INFO scheduling health checks: 2019-10-31 04:50:32,874 INFO scheduling jobs: 2019-10-31 04:50:32,876 DEBUG - qemu 2019-10-31 04:50:32,948 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:50:32,948 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:50:32,953 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:50:34,046 INFO [INIT] Dropping privileges 2019-10-31 04:50:34,046 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:50:34,046 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:50:34,048 INFO [INIT] Marking all workers as offline 2019-10-31 04:50:34,087 INFO [INIT] LAVA master has started. 2019-10-31 04:50:34,088 INFO [INIT] Using protocol version 3 2019-10-31 04:50:37,232 DEBUG Dhanu => PING(20) 2019-10-31 04:50:37,232 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:50:42,265 DEBUG lava-logs => PING(20) 2019-10-31 04:50:54,099 INFO scheduling health checks: 2019-10-31 04:50:54,131 INFO scheduling jobs: 2019-10-31 04:50:54,133 DEBUG - qemu 2019-10-31 04:50:54,201 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:50:54,201 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:50:54,209 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:50:55,276 INFO [INIT] Dropping privileges 2019-10-31 04:50:55,276 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:50:55,276 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:50:55,278 INFO [INIT] Marking all workers as offline 2019-10-31 04:50:55,312 INFO [INIT] LAVA master has started. 2019-10-31 04:50:55,312 INFO [INIT] Using protocol version 3 2019-10-31 04:50:57,252 DEBUG Dhanu => PING(20) 2019-10-31 04:50:57,253 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:51:02,285 DEBUG lava-logs => PING(20) 2019-10-31 04:51:15,319 INFO scheduling health checks: 2019-10-31 04:51:15,355 INFO scheduling jobs: 2019-10-31 04:51:15,358 DEBUG - qemu 2019-10-31 04:51:15,431 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:51:15,432 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:51:15,436 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:51:16,522 INFO [INIT] Dropping privileges 2019-10-31 04:51:16,523 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:51:16,523 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:51:16,524 INFO [INIT] Marking all workers as offline 2019-10-31 04:51:16,558 INFO [INIT] LAVA master has started. 2019-10-31 04:51:16,558 INFO [INIT] Using protocol version 3 2019-10-31 04:51:17,273 DEBUG Dhanu => PING(20) 2019-10-31 04:51:17,273 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:51:22,303 DEBUG lava-logs => PING(20) 2019-10-31 04:51:36,573 INFO scheduling health checks: 2019-10-31 04:51:36,601 INFO scheduling jobs: 2019-10-31 04:51:36,603 DEBUG - qemu 2019-10-31 04:51:36,674 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:51:36,674 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:51:36,679 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:51:37,793 INFO [INIT] Dropping privileges 2019-10-31 04:51:37,793 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:51:37,793 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:51:37,795 INFO [INIT] Marking all workers as offline 2019-10-31 04:51:37,837 INFO [INIT] LAVA master has started. 2019-10-31 04:51:37,837 INFO [INIT] Using protocol version 3 2019-10-31 04:51:37,838 DEBUG Dhanu => PING(20) 2019-10-31 04:51:37,838 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:51:42,321 DEBUG lava-logs => PING(20) 2019-10-31 04:51:57,853 INFO scheduling health checks: 2019-10-31 04:51:57,881 INFO scheduling jobs: 2019-10-31 04:51:57,883 DEBUG - qemu 2019-10-31 04:51:57,956 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:51:57,956 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:51:57,961 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:51:59,042 INFO [INIT] Dropping privileges 2019-10-31 04:51:59,042 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:51:59,042 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:51:59,044 INFO [INIT] Marking all workers as offline 2019-10-31 04:51:59,084 INFO [INIT] LAVA master has started. 2019-10-31 04:51:59,084 INFO [INIT] Using protocol version 3 2019-10-31 04:52:02,372 DEBUG lava-logs => PING(20) 2019-10-31 04:52:17,882 DEBUG Dhanu => PING(20) 2019-10-31 04:52:17,882 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:52:19,086 INFO scheduling health checks: 2019-10-31 04:52:19,139 INFO scheduling jobs: 2019-10-31 04:52:19,140 DEBUG - qemu 2019-10-31 04:52:19,207 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:52:19,208 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:52:19,217 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:52:20,285 INFO [INIT] Dropping privileges 2019-10-31 04:52:20,285 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:52:20,285 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:52:20,287 INFO [INIT] Marking all workers as offline 2019-10-31 04:52:20,352 INFO [INIT] LAVA master has started. 2019-10-31 04:52:20,352 INFO [INIT] Using protocol version 3 2019-10-31 04:52:22,387 DEBUG lava-logs => PING(20) 2019-10-31 04:52:37,903 DEBUG Dhanu => PING(20) 2019-10-31 04:52:37,904 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:52:40,355 INFO scheduling health checks: 2019-10-31 04:52:40,374 INFO scheduling jobs: 2019-10-31 04:52:40,375 DEBUG - qemu 2019-10-31 04:52:40,445 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:52:40,446 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:52:40,460 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:52:41,531 INFO [INIT] Dropping privileges 2019-10-31 04:52:41,531 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:52:41,531 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:52:41,533 INFO [INIT] Marking all workers as offline 2019-10-31 04:52:41,565 INFO [INIT] LAVA master has started. 2019-10-31 04:52:41,566 INFO [INIT] Using protocol version 3 2019-10-31 04:52:42,401 DEBUG lava-logs => PING(20) 2019-10-31 04:52:57,920 DEBUG Dhanu => PING(20) 2019-10-31 04:52:57,921 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:53:01,570 INFO scheduling health checks: 2019-10-31 04:53:01,621 INFO scheduling jobs: 2019-10-31 04:53:01,624 DEBUG - qemu 2019-10-31 04:53:01,695 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:53:01,695 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:53:01,700 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:53:02,776 INFO [INIT] Dropping privileges 2019-10-31 04:53:02,777 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:53:02,777 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:53:02,778 INFO [INIT] Marking all workers as offline 2019-10-31 04:53:02,812 INFO [INIT] LAVA master has started. 2019-10-31 04:53:02,813 INFO [INIT] Using protocol version 3 2019-10-31 04:53:02,918 DEBUG lava-logs => PING(20) 2019-10-31 04:53:17,939 DEBUG Dhanu => PING(20) 2019-10-31 04:53:17,940 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:53:22,818 INFO scheduling health checks: 2019-10-31 04:53:22,855 INFO scheduling jobs: 2019-10-31 04:53:22,857 DEBUG - qemu 2019-10-31 04:53:22,924 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:53:22,924 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:53:22,929 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:53:24,034 INFO [INIT] Dropping privileges 2019-10-31 04:53:24,034 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:53:24,034 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:53:24,036 INFO [INIT] Marking all workers as offline 2019-10-31 04:53:24,069 INFO [INIT] LAVA master has started. 2019-10-31 04:53:24,069 INFO [INIT] Using protocol version 3 2019-10-31 04:53:24,088 DEBUG lava-logs => PING(20) 2019-10-31 04:53:37,958 DEBUG Dhanu => PING(20) 2019-10-31 04:53:37,958 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:53:44,075 INFO scheduling health checks: 2019-10-31 04:53:44,113 INFO scheduling jobs: 2019-10-31 04:53:44,116 DEBUG - qemu 2019-10-31 04:53:44,184 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:53:44,184 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:53:44,189 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:53:45,293 INFO [INIT] Dropping privileges 2019-10-31 04:53:45,293 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:53:45,294 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:53:45,295 INFO [INIT] Marking all workers as offline 2019-10-31 04:53:45,326 INFO [INIT] LAVA master has started. 2019-10-31 04:53:45,326 INFO [INIT] Using protocol version 3 2019-10-31 04:53:57,980 DEBUG Dhanu => PING(20) 2019-10-31 04:53:57,980 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:54:04,131 DEBUG lava-logs => PING(20) 2019-10-31 04:54:05,328 INFO scheduling health checks: 2019-10-31 04:54:05,348 INFO scheduling jobs: 2019-10-31 04:54:05,349 DEBUG - qemu 2019-10-31 04:54:05,414 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:54:05,414 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:54:05,424 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:54:06,530 INFO [INIT] Dropping privileges 2019-10-31 04:54:06,530 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:54:06,530 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:54:06,532 INFO [INIT] Marking all workers as offline 2019-10-31 04:54:06,562 INFO [INIT] LAVA master has started. 2019-10-31 04:54:06,562 INFO [INIT] Using protocol version 3 2019-10-31 04:54:18,001 DEBUG Dhanu => PING(20) 2019-10-31 04:54:18,001 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:54:24,151 DEBUG lava-logs => PING(20) 2019-10-31 04:54:26,565 INFO scheduling health checks: 2019-10-31 04:54:26,583 INFO scheduling jobs: 2019-10-31 04:54:26,584 DEBUG - qemu 2019-10-31 04:54:26,646 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:54:26,646 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:54:26,651 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:54:27,786 INFO [INIT] Dropping privileges 2019-10-31 04:54:27,786 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:54:27,786 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:54:27,788 INFO [INIT] Marking all workers as offline 2019-10-31 04:54:27,820 INFO [INIT] LAVA master has started. 2019-10-31 04:54:27,820 INFO [INIT] Using protocol version 3 2019-10-31 04:54:38,020 DEBUG Dhanu => PING(20) 2019-10-31 04:54:38,021 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:54:44,171 DEBUG lava-logs => PING(20) 2019-10-31 04:54:47,824 INFO scheduling health checks: 2019-10-31 04:54:47,852 INFO scheduling jobs: 2019-10-31 04:54:47,854 DEBUG - qemu 2019-10-31 04:54:47,922 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:54:47,922 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:54:47,932 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:54:49,039 INFO [INIT] Dropping privileges 2019-10-31 04:54:49,039 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:54:49,039 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:54:49,040 INFO [INIT] Marking all workers as offline 2019-10-31 04:54:49,077 INFO [INIT] LAVA master has started. 2019-10-31 04:54:49,077 INFO [INIT] Using protocol version 3 2019-10-31 04:54:58,043 DEBUG Dhanu => PING(20) 2019-10-31 04:54:58,044 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:55:04,189 DEBUG lava-logs => PING(20) 2019-10-31 04:55:09,082 INFO scheduling health checks: 2019-10-31 04:55:09,110 INFO scheduling jobs: 2019-10-31 04:55:09,112 DEBUG - qemu 2019-10-31 04:55:09,185 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:55:09,185 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:55:09,190 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:55:10,281 INFO [INIT] Dropping privileges 2019-10-31 04:55:10,281 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:55:10,281 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:55:10,283 INFO [INIT] Marking all workers as offline 2019-10-31 04:55:10,322 INFO [INIT] LAVA master has started. 2019-10-31 04:55:10,323 INFO [INIT] Using protocol version 3 2019-10-31 04:55:18,064 DEBUG Dhanu => PING(20) 2019-10-31 04:55:18,065 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:55:24,206 DEBUG lava-logs => PING(20) 2019-10-31 04:55:30,329 INFO scheduling health checks: 2019-10-31 04:55:30,356 INFO scheduling jobs: 2019-10-31 04:55:30,358 DEBUG - qemu 2019-10-31 04:55:30,431 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:55:30,431 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:55:30,436 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:55:31,539 INFO [INIT] Dropping privileges 2019-10-31 04:55:31,539 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:55:31,539 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:55:31,541 INFO [INIT] Marking all workers as offline 2019-10-31 04:55:31,570 INFO [INIT] LAVA master has started. 2019-10-31 04:55:31,570 INFO [INIT] Using protocol version 3 2019-10-31 04:55:38,084 DEBUG Dhanu => PING(20) 2019-10-31 04:55:38,085 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:55:44,221 DEBUG lava-logs => PING(20) 2019-10-31 04:55:51,575 INFO scheduling health checks: 2019-10-31 04:55:51,614 INFO scheduling jobs: 2019-10-31 04:55:51,615 DEBUG - qemu 2019-10-31 04:55:51,687 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:55:51,687 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:55:51,697 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:55:52,822 INFO [INIT] Dropping privileges 2019-10-31 04:55:52,822 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:55:52,822 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:55:52,824 INFO [INIT] Marking all workers as offline 2019-10-31 04:55:52,849 INFO [INIT] LAVA master has started. 2019-10-31 04:55:52,850 INFO [INIT] Using protocol version 3 2019-10-31 04:55:58,104 DEBUG Dhanu => PING(20) 2019-10-31 04:55:58,104 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:56:04,234 DEBUG lava-logs => PING(20) 2019-10-31 04:56:12,859 INFO scheduling health checks: 2019-10-31 04:56:12,893 INFO scheduling jobs: 2019-10-31 04:56:12,896 DEBUG - qemu 2019-10-31 04:56:12,969 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:56:12,969 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:56:12,973 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:56:14,029 INFO [INIT] Dropping privileges 2019-10-31 04:56:14,029 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:56:14,029 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:56:14,031 INFO [INIT] Marking all workers as offline 2019-10-31 04:56:14,062 INFO [INIT] LAVA master has started. 2019-10-31 04:56:14,062 INFO [INIT] Using protocol version 3 2019-10-31 04:56:18,124 DEBUG Dhanu => PING(20) 2019-10-31 04:56:18,124 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:56:24,244 DEBUG lava-logs => PING(20) 2019-10-31 04:56:34,072 INFO scheduling health checks: 2019-10-31 04:56:34,106 INFO scheduling jobs: 2019-10-31 04:56:34,109 DEBUG - qemu 2019-10-31 04:56:34,182 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:56:34,182 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:56:34,187 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:56:35,313 INFO [INIT] Dropping privileges 2019-10-31 04:56:35,314 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:56:35,314 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:56:35,315 INFO [INIT] Marking all workers as offline 2019-10-31 04:56:35,376 INFO [INIT] LAVA master has started. 2019-10-31 04:56:35,376 INFO [INIT] Using protocol version 3 2019-10-31 04:56:38,145 DEBUG Dhanu => PING(20) 2019-10-31 04:56:38,146 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:56:44,264 DEBUG lava-logs => PING(20) 2019-10-31 04:56:55,387 INFO scheduling health checks: 2019-10-31 04:56:55,451 INFO scheduling jobs: 2019-10-31 04:56:55,455 DEBUG - qemu 2019-10-31 04:56:55,535 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:56:55,535 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:56:55,540 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:56:56,804 INFO [INIT] Dropping privileges 2019-10-31 04:56:56,804 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:56:56,804 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:56:56,806 INFO [INIT] Marking all workers as offline 2019-10-31 04:56:56,844 INFO [INIT] LAVA master has started. 2019-10-31 04:56:56,844 INFO [INIT] Using protocol version 3 2019-10-31 04:56:58,166 DEBUG Dhanu => PING(20) 2019-10-31 04:56:58,166 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:57:04,272 DEBUG lava-logs => PING(20) 2019-10-31 04:57:16,855 INFO scheduling health checks: 2019-10-31 04:57:16,888 INFO scheduling jobs: 2019-10-31 04:57:16,889 DEBUG - qemu 2019-10-31 04:57:16,976 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:57:16,976 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:57:16,982 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:57:18,317 INFO [INIT] Dropping privileges 2019-10-31 04:57:18,317 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:57:18,317 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:57:18,318 INFO [INIT] Marking all workers as offline 2019-10-31 04:57:18,379 INFO [INIT] LAVA master has started. 2019-10-31 04:57:18,379 INFO [INIT] Using protocol version 3 2019-10-31 04:57:18,505 DEBUG Dhanu => PING(20) 2019-10-31 04:57:18,506 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:57:24,288 DEBUG lava-logs => PING(20) 2019-10-31 04:57:38,394 INFO scheduling health checks: 2019-10-31 04:57:38,435 INFO scheduling jobs: 2019-10-31 04:57:38,438 DEBUG - qemu 2019-10-31 04:57:38,534 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:57:38,534 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:57:38,541 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:57:39,951 INFO [INIT] Dropping privileges 2019-10-31 04:57:39,952 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:57:39,952 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:57:39,953 INFO [INIT] Marking all workers as offline 2019-10-31 04:57:40,003 INFO [INIT] LAVA master has started. 2019-10-31 04:57:40,004 INFO [INIT] Using protocol version 3 2019-10-31 04:57:44,305 DEBUG lava-logs => PING(20) 2019-10-31 04:57:58,540 DEBUG Dhanu => PING(20) 2019-10-31 04:57:58,541 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:58:00,006 INFO scheduling health checks: 2019-10-31 04:58:00,025 INFO scheduling jobs: 2019-10-31 04:58:00,026 DEBUG - qemu 2019-10-31 04:58:00,103 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:58:00,103 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:58:00,107 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:58:01,390 INFO [INIT] Dropping privileges 2019-10-31 04:58:01,390 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:58:01,390 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:58:01,392 INFO [INIT] Marking all workers as offline 2019-10-31 04:58:01,428 INFO [INIT] LAVA master has started. 2019-10-31 04:58:01,429 INFO [INIT] Using protocol version 3 2019-10-31 04:58:04,319 DEBUG lava-logs => PING(20) 2019-10-31 04:58:18,560 DEBUG Dhanu => PING(20) 2019-10-31 04:58:18,560 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:58:21,431 INFO scheduling health checks: 2019-10-31 04:58:21,517 INFO scheduling jobs: 2019-10-31 04:58:21,520 DEBUG - qemu 2019-10-31 04:58:21,595 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:58:21,595 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:58:21,600 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:58:22,813 INFO [INIT] Dropping privileges 2019-10-31 04:58:22,814 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:58:22,814 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:58:22,815 INFO [INIT] Marking all workers as offline 2019-10-31 04:58:22,841 INFO [INIT] LAVA master has started. 2019-10-31 04:58:22,841 INFO [INIT] Using protocol version 3 2019-10-31 04:58:24,332 DEBUG lava-logs => PING(20) 2019-10-31 04:58:38,576 DEBUG Dhanu => PING(20) 2019-10-31 04:58:38,577 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:58:42,846 INFO scheduling health checks: 2019-10-31 04:58:42,885 INFO scheduling jobs: 2019-10-31 04:58:42,887 DEBUG - qemu 2019-10-31 04:58:42,954 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:58:42,955 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:58:42,959 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:58:44,318 INFO [INIT] Dropping privileges 2019-10-31 04:58:44,318 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:58:44,318 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:58:44,319 INFO [INIT] Marking all workers as offline 2019-10-31 04:58:44,364 INFO [INIT] LAVA master has started. 2019-10-31 04:58:44,365 INFO [INIT] Using protocol version 3 2019-10-31 04:58:44,507 DEBUG lava-logs => PING(20) 2019-10-31 04:58:58,594 DEBUG Dhanu => PING(20) 2019-10-31 04:58:58,595 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:59:04,369 INFO scheduling health checks: 2019-10-31 04:59:04,420 INFO scheduling jobs: 2019-10-31 04:59:04,423 DEBUG - qemu 2019-10-31 04:59:04,500 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:59:04,500 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:59:04,505 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:59:06,111 INFO [INIT] Dropping privileges 2019-10-31 04:59:06,112 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:59:06,112 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:59:06,113 INFO [INIT] Marking all workers as offline 2019-10-31 04:59:06,144 INFO [INIT] LAVA master has started. 2019-10-31 04:59:06,144 INFO [INIT] Using protocol version 3 2019-10-31 04:59:06,300 DEBUG lava-logs => PING(20) 2019-10-31 04:59:18,612 DEBUG Dhanu => PING(20) 2019-10-31 04:59:18,613 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:59:26,151 INFO scheduling health checks: 2019-10-31 04:59:26,178 INFO scheduling jobs: 2019-10-31 04:59:26,180 DEBUG - qemu 2019-10-31 04:59:26,272 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:59:26,272 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:59:26,281 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:59:27,660 INFO [INIT] Dropping privileges 2019-10-31 04:59:27,660 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:59:27,660 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:59:27,662 INFO [INIT] Marking all workers as offline 2019-10-31 04:59:27,691 INFO [INIT] LAVA master has started. 2019-10-31 04:59:27,691 INFO [INIT] Using protocol version 3 2019-10-31 04:59:27,870 DEBUG lava-logs => PING(20) 2019-10-31 04:59:38,624 DEBUG Dhanu => PING(20) 2019-10-31 04:59:38,624 WARNING Unknown dispatcher (server crashed) 2019-10-31 04:59:47,700 INFO scheduling health checks: 2019-10-31 04:59:47,735 INFO scheduling jobs: 2019-10-31 04:59:47,738 DEBUG - qemu 2019-10-31 04:59:47,806 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 04:59:47,807 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 04:59:47,811 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 04:59:49,297 INFO [INIT] Dropping privileges 2019-10-31 04:59:49,297 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 04:59:49,298 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 04:59:49,299 INFO [INIT] Marking all workers as offline 2019-10-31 04:59:49,339 INFO [INIT] LAVA master has started. 2019-10-31 04:59:49,339 INFO [INIT] Using protocol version 3 2019-10-31 04:59:49,382 DEBUG lava-logs => PING(20) 2019-10-31 04:59:58,640 DEBUG Dhanu => PING(20) 2019-10-31 04:59:58,641 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:00:09,351 INFO scheduling health checks: 2019-10-31 05:00:09,393 INFO scheduling jobs: 2019-10-31 05:00:09,397 DEBUG - qemu 2019-10-31 05:00:09,475 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:00:09,475 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:00:09,482 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:00:10,813 INFO [INIT] Dropping privileges 2019-10-31 05:00:10,813 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:00:10,813 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:00:10,815 INFO [INIT] Marking all workers as offline 2019-10-31 05:00:10,850 INFO [INIT] LAVA master has started. 2019-10-31 05:00:10,850 INFO [INIT] Using protocol version 3 2019-10-31 05:00:18,660 DEBUG Dhanu => PING(20) 2019-10-31 05:00:18,661 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:00:29,416 DEBUG lava-logs => PING(20) 2019-10-31 05:00:30,851 INFO scheduling health checks: 2019-10-31 05:00:30,895 INFO scheduling jobs: 2019-10-31 05:00:30,898 DEBUG - qemu 2019-10-31 05:00:30,973 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:00:30,974 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:00:30,984 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:00:32,310 INFO [INIT] Dropping privileges 2019-10-31 05:00:32,310 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:00:32,310 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:00:32,312 INFO [INIT] Marking all workers as offline 2019-10-31 05:00:32,341 INFO [INIT] LAVA master has started. 2019-10-31 05:00:32,342 INFO [INIT] Using protocol version 3 2019-10-31 05:00:38,671 DEBUG Dhanu => PING(20) 2019-10-31 05:00:38,672 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:00:49,434 DEBUG lava-logs => PING(20) 2019-10-31 05:00:52,343 INFO scheduling health checks: 2019-10-31 05:00:52,374 INFO scheduling jobs: 2019-10-31 05:00:52,376 DEBUG - qemu 2019-10-31 05:00:52,461 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:00:52,461 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:00:52,465 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:00:53,913 INFO [INIT] Dropping privileges 2019-10-31 05:00:53,913 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:00:53,913 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:00:53,915 INFO [INIT] Marking all workers as offline 2019-10-31 05:00:53,954 INFO [INIT] LAVA master has started. 2019-10-31 05:00:53,954 INFO [INIT] Using protocol version 3 2019-10-31 05:00:58,689 DEBUG Dhanu => PING(20) 2019-10-31 05:00:58,690 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:01:09,450 DEBUG lava-logs => PING(20) 2019-10-31 05:01:13,959 INFO scheduling health checks: 2019-10-31 05:01:13,988 INFO scheduling jobs: 2019-10-31 05:01:13,989 DEBUG - qemu 2019-10-31 05:01:14,057 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:01:14,057 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:01:14,062 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:01:15,292 INFO [INIT] Dropping privileges 2019-10-31 05:01:15,292 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:01:15,292 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:01:15,294 INFO [INIT] Marking all workers as offline 2019-10-31 05:01:15,335 INFO [INIT] LAVA master has started. 2019-10-31 05:01:15,335 INFO [INIT] Using protocol version 3 2019-10-31 05:01:18,711 DEBUG Dhanu => PING(20) 2019-10-31 05:01:18,711 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:01:29,465 DEBUG lava-logs => PING(20) 2019-10-31 05:01:35,339 INFO scheduling health checks: 2019-10-31 05:01:35,379 INFO scheduling jobs: 2019-10-31 05:01:35,381 DEBUG - qemu 2019-10-31 05:01:35,459 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:01:35,459 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:01:35,466 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:01:36,653 INFO [INIT] Dropping privileges 2019-10-31 05:01:36,654 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:01:36,654 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:01:36,655 INFO [INIT] Marking all workers as offline 2019-10-31 05:01:36,680 INFO [INIT] LAVA master has started. 2019-10-31 05:01:36,681 INFO [INIT] Using protocol version 3 2019-10-31 05:01:38,732 DEBUG Dhanu => PING(20) 2019-10-31 05:01:38,733 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:01:49,478 DEBUG lava-logs => PING(20) 2019-10-31 05:01:56,687 INFO scheduling health checks: 2019-10-31 05:01:56,725 INFO scheduling jobs: 2019-10-31 05:01:56,726 DEBUG - qemu 2019-10-31 05:01:56,799 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:01:56,799 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:01:56,804 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:01:58,070 INFO [INIT] Dropping privileges 2019-10-31 05:01:58,070 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:01:58,071 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:01:58,072 INFO [INIT] Marking all workers as offline 2019-10-31 05:01:58,105 INFO [INIT] LAVA master has started. 2019-10-31 05:01:58,105 INFO [INIT] Using protocol version 3 2019-10-31 05:01:58,747 DEBUG Dhanu => PING(20) 2019-10-31 05:01:58,747 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:02:09,494 DEBUG lava-logs => PING(20) 2019-10-31 05:02:18,114 INFO scheduling health checks: 2019-10-31 05:02:18,160 INFO scheduling jobs: 2019-10-31 05:02:18,164 DEBUG - qemu 2019-10-31 05:02:18,244 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:02:18,244 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:02:18,254 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:02:19,652 INFO [INIT] Dropping privileges 2019-10-31 05:02:19,652 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:02:19,652 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:02:19,654 INFO [INIT] Marking all workers as offline 2019-10-31 05:02:19,699 INFO [INIT] LAVA master has started. 2019-10-31 05:02:19,699 INFO [INIT] Using protocol version 3 2019-10-31 05:02:19,731 DEBUG Dhanu => PING(20) 2019-10-31 05:02:19,732 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:02:29,517 DEBUG lava-logs => PING(20) 2019-10-31 05:02:39,707 INFO scheduling health checks: 2019-10-31 05:02:39,740 INFO scheduling jobs: 2019-10-31 05:02:39,744 DEBUG - qemu 2019-10-31 05:02:39,832 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:02:39,832 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:02:39,837 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:02:41,087 INFO [INIT] Dropping privileges 2019-10-31 05:02:41,087 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:02:41,087 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:02:41,089 INFO [INIT] Marking all workers as offline 2019-10-31 05:02:41,132 INFO [INIT] LAVA master has started. 2019-10-31 05:02:41,133 INFO [INIT] Using protocol version 3 2019-10-31 05:02:49,536 DEBUG lava-logs => PING(20) 2019-10-31 05:02:59,764 DEBUG Dhanu => PING(20) 2019-10-31 05:02:59,765 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:03:01,135 INFO scheduling health checks: 2019-10-31 05:03:01,165 INFO scheduling jobs: 2019-10-31 05:03:01,167 DEBUG - qemu 2019-10-31 05:03:01,238 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:03:01,239 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:03:01,243 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:03:02,794 INFO [INIT] Dropping privileges 2019-10-31 05:03:02,794 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:03:02,794 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:03:02,796 INFO [INIT] Marking all workers as offline 2019-10-31 05:03:02,844 INFO [INIT] LAVA master has started. 2019-10-31 05:03:02,845 INFO [INIT] Using protocol version 3 2019-10-31 05:03:09,547 DEBUG lava-logs => PING(20) 2019-10-31 05:03:19,781 DEBUG Dhanu => PING(20) 2019-10-31 05:03:19,782 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:03:22,847 INFO scheduling health checks: 2019-10-31 05:03:22,866 INFO scheduling jobs: 2019-10-31 05:03:22,868 DEBUG - qemu 2019-10-31 05:03:22,945 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:03:22,946 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:03:22,951 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:03:24,188 INFO [INIT] Dropping privileges 2019-10-31 05:03:24,188 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:03:24,188 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:03:24,190 INFO [INIT] Marking all workers as offline 2019-10-31 05:03:24,236 INFO [INIT] LAVA master has started. 2019-10-31 05:03:24,237 INFO [INIT] Using protocol version 3 2019-10-31 05:03:29,562 DEBUG lava-logs => PING(20) 2019-10-31 05:03:39,796 DEBUG Dhanu => PING(20) 2019-10-31 05:03:39,796 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:03:44,239 INFO scheduling health checks: 2019-10-31 05:03:44,280 INFO scheduling jobs: 2019-10-31 05:03:44,283 DEBUG - qemu 2019-10-31 05:03:44,386 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:03:44,386 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:03:44,392 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:03:45,706 INFO [INIT] Dropping privileges 2019-10-31 05:03:45,706 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:03:45,707 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:03:45,708 INFO [INIT] Marking all workers as offline 2019-10-31 05:03:45,749 INFO [INIT] LAVA master has started. 2019-10-31 05:03:45,750 INFO [INIT] Using protocol version 3 2019-10-31 05:03:49,575 DEBUG lava-logs => PING(20) 2019-10-31 05:03:59,812 DEBUG Dhanu => PING(20) 2019-10-31 05:03:59,813 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:04:05,755 INFO scheduling health checks: 2019-10-31 05:04:05,793 INFO scheduling jobs: 2019-10-31 05:04:05,795 DEBUG - qemu 2019-10-31 05:04:05,875 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:04:05,875 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:04:05,879 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:04:07,039 INFO [INIT] Dropping privileges 2019-10-31 05:04:07,039 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:04:07,039 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:04:07,041 INFO [INIT] Marking all workers as offline 2019-10-31 05:04:07,085 INFO [INIT] LAVA master has started. 2019-10-31 05:04:07,085 INFO [INIT] Using protocol version 3 2019-10-31 05:04:09,587 DEBUG lava-logs => PING(20) 2019-10-31 05:04:19,829 DEBUG Dhanu => PING(20) 2019-10-31 05:04:19,830 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:04:27,093 INFO scheduling health checks: 2019-10-31 05:04:27,128 INFO scheduling jobs: 2019-10-31 05:04:27,132 DEBUG - qemu 2019-10-31 05:04:27,206 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:04:27,207 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:04:27,211 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:04:28,285 INFO [INIT] Dropping privileges 2019-10-31 05:04:28,286 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:04:28,286 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:04:28,287 INFO [INIT] Marking all workers as offline 2019-10-31 05:04:28,318 INFO [INIT] LAVA master has started. 2019-10-31 05:04:28,318 INFO [INIT] Using protocol version 3 2019-10-31 05:04:29,600 DEBUG lava-logs => PING(20) 2019-10-31 05:04:39,850 DEBUG Dhanu => PING(20) 2019-10-31 05:04:39,851 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:04:48,327 INFO scheduling health checks: 2019-10-31 05:04:48,364 INFO scheduling jobs: 2019-10-31 05:04:48,366 DEBUG - qemu 2019-10-31 05:04:48,440 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:04:48,440 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:04:48,445 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:04:49,534 INFO [INIT] Dropping privileges 2019-10-31 05:04:49,535 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:04:49,535 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:04:49,536 INFO [INIT] Marking all workers as offline 2019-10-31 05:04:49,578 INFO [INIT] LAVA master has started. 2019-10-31 05:04:49,578 INFO [INIT] Using protocol version 3 2019-10-31 05:04:49,686 DEBUG lava-logs => PING(20) 2019-10-31 05:04:59,864 DEBUG Dhanu => PING(20) 2019-10-31 05:04:59,865 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:05:09,589 INFO scheduling health checks: 2019-10-31 05:05:09,621 INFO scheduling jobs: 2019-10-31 05:05:09,623 DEBUG - qemu 2019-10-31 05:05:09,698 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:05:09,698 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:05:09,703 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:05:10,793 INFO [INIT] Dropping privileges 2019-10-31 05:05:10,793 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:05:10,794 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:05:10,795 INFO [INIT] Marking all workers as offline 2019-10-31 05:05:10,824 INFO [INIT] LAVA master has started. 2019-10-31 05:05:10,824 INFO [INIT] Using protocol version 3 2019-10-31 05:05:10,838 DEBUG lava-logs => PING(20) 2019-10-31 05:05:19,886 DEBUG Dhanu => PING(20) 2019-10-31 05:05:19,887 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:05:30,831 INFO scheduling health checks: 2019-10-31 05:05:30,867 INFO scheduling jobs: 2019-10-31 05:05:30,869 DEBUG - qemu 2019-10-31 05:05:30,948 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:05:30,948 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:05:30,953 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:05:32,024 INFO [INIT] Dropping privileges 2019-10-31 05:05:32,024 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:05:32,024 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:05:32,026 INFO [INIT] Marking all workers as offline 2019-10-31 05:05:32,059 INFO [INIT] LAVA master has started. 2019-10-31 05:05:32,060 INFO [INIT] Using protocol version 3 2019-10-31 05:05:39,909 DEBUG Dhanu => PING(20) 2019-10-31 05:05:39,910 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:05:50,881 DEBUG lava-logs => PING(20) 2019-10-31 05:05:52,061 INFO scheduling health checks: 2019-10-31 05:05:52,092 INFO scheduling jobs: 2019-10-31 05:05:52,094 DEBUG - qemu 2019-10-31 05:05:52,166 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:05:52,166 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:05:52,171 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:05:53,285 INFO [INIT] Dropping privileges 2019-10-31 05:05:53,285 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:05:53,285 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:05:53,287 INFO [INIT] Marking all workers as offline 2019-10-31 05:05:53,317 INFO [INIT] LAVA master has started. 2019-10-31 05:05:53,317 INFO [INIT] Using protocol version 3 2019-10-31 05:05:59,931 DEBUG Dhanu => PING(20) 2019-10-31 05:05:59,932 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:06:10,902 DEBUG lava-logs => PING(20) 2019-10-31 05:06:13,320 INFO scheduling health checks: 2019-10-31 05:06:13,361 INFO scheduling jobs: 2019-10-31 05:06:13,364 DEBUG - qemu 2019-10-31 05:06:13,438 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:06:13,438 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:06:13,442 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:06:14,529 INFO [INIT] Dropping privileges 2019-10-31 05:06:14,529 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:06:14,529 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:06:14,531 INFO [INIT] Marking all workers as offline 2019-10-31 05:06:14,563 INFO [INIT] LAVA master has started. 2019-10-31 05:06:14,564 INFO [INIT] Using protocol version 3 2019-10-31 05:06:19,950 DEBUG Dhanu => PING(20) 2019-10-31 05:06:19,950 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:06:30,919 DEBUG lava-logs => PING(20) 2019-10-31 05:06:34,567 INFO scheduling health checks: 2019-10-31 05:06:34,641 INFO scheduling jobs: 2019-10-31 05:06:34,643 DEBUG - qemu 2019-10-31 05:06:34,713 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:06:34,713 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:06:34,718 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:06:35,787 INFO [INIT] Dropping privileges 2019-10-31 05:06:35,788 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:06:35,788 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:06:35,789 INFO [INIT] Marking all workers as offline 2019-10-31 05:06:35,821 INFO [INIT] LAVA master has started. 2019-10-31 05:06:35,821 INFO [INIT] Using protocol version 3 2019-10-31 05:06:39,971 DEBUG Dhanu => PING(20) 2019-10-31 05:06:39,971 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:06:50,937 DEBUG lava-logs => PING(20) 2019-10-31 05:06:55,827 INFO scheduling health checks: 2019-10-31 05:06:55,864 INFO scheduling jobs: 2019-10-31 05:06:55,866 DEBUG - qemu 2019-10-31 05:06:55,933 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:06:55,934 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:06:55,938 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:06:57,032 INFO [INIT] Dropping privileges 2019-10-31 05:06:57,032 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:06:57,033 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:06:57,034 INFO [INIT] Marking all workers as offline 2019-10-31 05:06:57,068 INFO [INIT] LAVA master has started. 2019-10-31 05:06:57,068 INFO [INIT] Using protocol version 3 2019-10-31 05:06:59,992 DEBUG Dhanu => PING(20) 2019-10-31 05:06:59,993 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:07:10,953 DEBUG lava-logs => PING(20) 2019-10-31 05:07:17,074 INFO scheduling health checks: 2019-10-31 05:07:17,111 INFO scheduling jobs: 2019-10-31 05:07:17,113 DEBUG - qemu 2019-10-31 05:07:17,187 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:07:17,187 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:07:17,191 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:07:18,279 INFO [INIT] Dropping privileges 2019-10-31 05:07:18,279 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:07:18,279 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:07:18,281 INFO [INIT] Marking all workers as offline 2019-10-31 05:07:18,314 INFO [INIT] LAVA master has started. 2019-10-31 05:07:18,314 INFO [INIT] Using protocol version 3 2019-10-31 05:07:20,010 DEBUG Dhanu => PING(20) 2019-10-31 05:07:20,010 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:07:30,967 DEBUG lava-logs => PING(20) 2019-10-31 05:07:38,322 INFO scheduling health checks: 2019-10-31 05:07:38,357 INFO scheduling jobs: 2019-10-31 05:07:38,360 DEBUG - qemu 2019-10-31 05:07:38,434 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:07:38,434 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:07:38,439 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:07:39,561 INFO [INIT] Dropping privileges 2019-10-31 05:07:39,561 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:07:39,561 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:07:39,563 INFO [INIT] Marking all workers as offline 2019-10-31 05:07:39,604 INFO [INIT] LAVA master has started. 2019-10-31 05:07:39,604 INFO [INIT] Using protocol version 3 2019-10-31 05:07:40,025 DEBUG Dhanu => PING(20) 2019-10-31 05:07:40,026 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:07:50,982 DEBUG lava-logs => PING(20) 2019-10-31 05:07:59,611 INFO scheduling health checks: 2019-10-31 05:07:59,648 INFO scheduling jobs: 2019-10-31 05:07:59,649 DEBUG - qemu 2019-10-31 05:07:59,768 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:07:59,768 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:07:59,778 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:08:01,162 INFO [INIT] Dropping privileges 2019-10-31 05:08:01,162 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:08:01,162 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:08:01,164 INFO [INIT] Marking all workers as offline 2019-10-31 05:08:01,207 INFO [INIT] LAVA master has started. 2019-10-31 05:08:01,207 INFO [INIT] Using protocol version 3 2019-10-31 05:08:01,279 DEBUG Dhanu => PING(20) 2019-10-31 05:08:01,280 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:08:11,002 DEBUG lava-logs => PING(20) 2019-10-31 05:08:21,215 INFO scheduling health checks: 2019-10-31 05:08:21,262 INFO scheduling jobs: 2019-10-31 05:08:21,264 DEBUG - qemu 2019-10-31 05:08:21,342 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:08:21,342 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:08:21,351 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:08:22,683 INFO [INIT] Dropping privileges 2019-10-31 05:08:22,683 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:08:22,683 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:08:22,685 INFO [INIT] Marking all workers as offline 2019-10-31 05:08:22,742 INFO [INIT] LAVA master has started. 2019-10-31 05:08:22,743 INFO [INIT] Using protocol version 3 2019-10-31 05:08:31,020 DEBUG lava-logs => PING(20) 2019-10-31 05:08:41,312 DEBUG Dhanu => PING(20) 2019-10-31 05:08:41,312 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:08:42,744 INFO scheduling health checks: 2019-10-31 05:08:42,763 INFO scheduling jobs: 2019-10-31 05:08:42,765 DEBUG - qemu 2019-10-31 05:08:42,837 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:08:42,837 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:08:42,842 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:08:44,136 INFO [INIT] Dropping privileges 2019-10-31 05:08:44,136 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:08:44,136 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:08:44,138 INFO [INIT] Marking all workers as offline 2019-10-31 05:08:44,176 INFO [INIT] LAVA master has started. 2019-10-31 05:08:44,176 INFO [INIT] Using protocol version 3 2019-10-31 05:08:51,036 DEBUG lava-logs => PING(20) 2019-10-31 05:09:01,328 DEBUG Dhanu => PING(20) 2019-10-31 05:09:01,329 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:09:04,179 INFO scheduling health checks: 2019-10-31 05:09:04,210 INFO scheduling jobs: 2019-10-31 05:09:04,212 DEBUG - qemu 2019-10-31 05:09:04,292 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:09:04,292 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:09:04,297 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:09:05,593 INFO [INIT] Dropping privileges 2019-10-31 05:09:05,593 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:09:05,593 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:09:05,595 INFO [INIT] Marking all workers as offline 2019-10-31 05:09:05,635 INFO [INIT] LAVA master has started. 2019-10-31 05:09:05,636 INFO [INIT] Using protocol version 3 2019-10-31 05:09:11,052 DEBUG lava-logs => PING(20) 2019-10-31 05:09:21,344 DEBUG Dhanu => PING(20) 2019-10-31 05:09:21,345 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:09:25,640 INFO scheduling health checks: 2019-10-31 05:09:25,679 INFO scheduling jobs: 2019-10-31 05:09:25,680 DEBUG - qemu 2019-10-31 05:09:25,749 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:09:25,749 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:09:25,754 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:09:27,037 INFO [INIT] Dropping privileges 2019-10-31 05:09:27,037 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:09:27,038 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:09:27,039 INFO [INIT] Marking all workers as offline 2019-10-31 05:09:27,070 INFO [INIT] LAVA master has started. 2019-10-31 05:09:27,070 INFO [INIT] Using protocol version 3 2019-10-31 05:09:31,069 DEBUG lava-logs => PING(20) 2019-10-31 05:09:41,367 DEBUG Dhanu => PING(20) 2019-10-31 05:09:41,368 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:09:47,076 INFO scheduling health checks: 2019-10-31 05:09:47,114 INFO scheduling jobs: 2019-10-31 05:09:47,117 DEBUG - qemu 2019-10-31 05:09:47,189 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:09:47,189 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:09:47,194 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:09:48,294 INFO [INIT] Dropping privileges 2019-10-31 05:09:48,295 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:09:48,295 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:09:48,296 INFO [INIT] Marking all workers as offline 2019-10-31 05:09:48,339 INFO [INIT] LAVA master has started. 2019-10-31 05:09:48,339 INFO [INIT] Using protocol version 3 2019-10-31 05:09:51,084 DEBUG lava-logs => PING(20) 2019-10-31 05:10:01,390 DEBUG Dhanu => PING(20) 2019-10-31 05:10:01,390 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:10:08,346 INFO scheduling health checks: 2019-10-31 05:10:08,383 INFO scheduling jobs: 2019-10-31 05:10:08,385 DEBUG - qemu 2019-10-31 05:10:08,452 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:10:08,452 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:10:08,457 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:10:09,534 INFO [INIT] Dropping privileges 2019-10-31 05:10:09,535 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:10:09,535 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:10:09,536 INFO [INIT] Marking all workers as offline 2019-10-31 05:10:09,597 INFO [INIT] LAVA master has started. 2019-10-31 05:10:09,597 INFO [INIT] Using protocol version 3 2019-10-31 05:10:11,099 DEBUG lava-logs => PING(20) 2019-10-31 05:10:21,410 DEBUG Dhanu => PING(20) 2019-10-31 05:10:21,410 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:10:29,599 INFO scheduling health checks: 2019-10-31 05:10:29,641 INFO scheduling jobs: 2019-10-31 05:10:29,643 DEBUG - qemu 2019-10-31 05:10:29,719 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:10:29,719 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:10:29,735 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:10:31,043 INFO [INIT] Dropping privileges 2019-10-31 05:10:31,043 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:10:31,043 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:10:31,045 INFO [INIT] Marking all workers as offline 2019-10-31 05:10:31,087 INFO [INIT] LAVA master has started. 2019-10-31 05:10:31,088 INFO [INIT] Using protocol version 3 2019-10-31 05:10:31,112 DEBUG lava-logs => PING(20) 2019-10-31 05:10:41,428 DEBUG Dhanu => PING(20) 2019-10-31 05:10:41,429 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:10:51,098 INFO scheduling health checks: 2019-10-31 05:10:51,131 INFO scheduling jobs: 2019-10-31 05:10:51,133 DEBUG - qemu 2019-10-31 05:10:51,205 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:10:51,205 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:10:51,210 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:10:52,278 INFO [INIT] Dropping privileges 2019-10-31 05:10:52,279 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:10:52,279 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:10:52,280 INFO [INIT] Marking all workers as offline 2019-10-31 05:10:52,312 INFO [INIT] LAVA master has started. 2019-10-31 05:10:52,312 INFO [INIT] Using protocol version 3 2019-10-31 05:11:01,448 DEBUG Dhanu => PING(20) 2019-10-31 05:11:01,449 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:11:11,154 DEBUG lava-logs => PING(20) 2019-10-31 05:11:12,314 INFO scheduling health checks: 2019-10-31 05:11:12,333 INFO scheduling jobs: 2019-10-31 05:11:12,335 DEBUG - qemu 2019-10-31 05:11:12,405 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:11:12,405 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:11:12,410 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:11:13,537 INFO [INIT] Dropping privileges 2019-10-31 05:11:13,538 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:11:13,538 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:11:13,539 INFO [INIT] Marking all workers as offline 2019-10-31 05:11:13,592 INFO [INIT] LAVA master has started. 2019-10-31 05:11:13,592 INFO [INIT] Using protocol version 3 2019-10-31 05:11:21,470 DEBUG Dhanu => PING(20) 2019-10-31 05:11:21,470 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:11:31,172 DEBUG lava-logs => PING(20) 2019-10-31 05:11:33,594 INFO scheduling health checks: 2019-10-31 05:11:33,635 INFO scheduling jobs: 2019-10-31 05:11:33,637 DEBUG - qemu 2019-10-31 05:11:33,712 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:11:33,712 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:11:33,717 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:11:34,803 INFO [INIT] Dropping privileges 2019-10-31 05:11:34,803 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:11:34,803 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:11:34,805 INFO [INIT] Marking all workers as offline 2019-10-31 05:11:34,838 INFO [INIT] LAVA master has started. 2019-10-31 05:11:34,838 INFO [INIT] Using protocol version 3 2019-10-31 05:11:41,488 DEBUG Dhanu => PING(20) 2019-10-31 05:11:41,488 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:11:51,189 DEBUG lava-logs => PING(20) 2019-10-31 05:11:54,840 INFO scheduling health checks: 2019-10-31 05:11:54,871 INFO scheduling jobs: 2019-10-31 05:11:54,874 DEBUG - qemu 2019-10-31 05:11:54,953 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:11:54,954 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:11:54,958 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:11:56,109 INFO [INIT] Dropping privileges 2019-10-31 05:11:56,109 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:11:56,109 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:11:56,111 INFO [INIT] Marking all workers as offline 2019-10-31 05:11:56,140 INFO [INIT] LAVA master has started. 2019-10-31 05:11:56,140 INFO [INIT] Using protocol version 3 2019-10-31 05:12:01,504 DEBUG Dhanu => PING(20) 2019-10-31 05:12:01,505 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:12:11,208 DEBUG lava-logs => PING(20) 2019-10-31 05:12:16,145 INFO scheduling health checks: 2019-10-31 05:12:16,184 INFO scheduling jobs: 2019-10-31 05:12:16,187 DEBUG - qemu 2019-10-31 05:12:16,262 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:12:16,262 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:12:16,266 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:12:17,793 INFO [INIT] Dropping privileges 2019-10-31 05:12:17,793 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:12:17,793 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:12:17,795 INFO [INIT] Marking all workers as offline 2019-10-31 05:12:17,820 INFO [INIT] LAVA master has started. 2019-10-31 05:12:17,820 INFO [INIT] Using protocol version 3 2019-10-31 05:12:21,524 DEBUG Dhanu => PING(20) 2019-10-31 05:12:21,525 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:12:31,221 DEBUG lava-logs => PING(20) 2019-10-31 05:12:37,827 INFO scheduling health checks: 2019-10-31 05:12:37,875 INFO scheduling jobs: 2019-10-31 05:12:37,878 DEBUG - qemu 2019-10-31 05:12:37,959 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:12:37,960 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:12:37,964 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:12:39,211 INFO [INIT] Dropping privileges 2019-10-31 05:12:39,211 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:12:39,211 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:12:39,213 INFO [INIT] Marking all workers as offline 2019-10-31 05:12:39,244 INFO [INIT] LAVA master has started. 2019-10-31 05:12:39,244 INFO [INIT] Using protocol version 3 2019-10-31 05:12:41,540 DEBUG Dhanu => PING(20) 2019-10-31 05:12:41,540 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:12:51,232 DEBUG lava-logs => PING(20) 2019-10-31 05:12:59,251 INFO scheduling health checks: 2019-10-31 05:12:59,277 INFO scheduling jobs: 2019-10-31 05:12:59,279 DEBUG - qemu 2019-10-31 05:12:59,365 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:12:59,366 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:12:59,370 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:13:00,564 INFO [INIT] Dropping privileges 2019-10-31 05:13:00,564 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:13:00,564 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:13:00,566 INFO [INIT] Marking all workers as offline 2019-10-31 05:13:00,602 INFO [INIT] LAVA master has started. 2019-10-31 05:13:00,602 INFO [INIT] Using protocol version 3 2019-10-31 05:13:01,554 DEBUG Dhanu => PING(20) 2019-10-31 05:13:01,554 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:13:11,244 DEBUG lava-logs => PING(20) 2019-10-31 05:13:20,612 INFO scheduling health checks: 2019-10-31 05:13:20,657 INFO scheduling jobs: 2019-10-31 05:13:20,660 DEBUG - qemu 2019-10-31 05:13:20,750 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:13:20,750 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:13:20,757 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:13:22,218 INFO [INIT] Dropping privileges 2019-10-31 05:13:22,219 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:13:22,219 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:13:22,221 INFO [INIT] Marking all workers as offline 2019-10-31 05:13:22,249 INFO [INIT] LAVA master has started. 2019-10-31 05:13:22,249 INFO [INIT] Using protocol version 3 2019-10-31 05:13:22,337 DEBUG Dhanu => PING(20) 2019-10-31 05:13:22,338 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:13:31,264 DEBUG lava-logs => PING(20) 2019-10-31 05:13:42,259 INFO scheduling health checks: 2019-10-31 05:13:42,292 INFO scheduling jobs: 2019-10-31 05:13:42,294 DEBUG - qemu 2019-10-31 05:13:42,366 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:13:42,366 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:13:42,370 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:13:43,552 INFO [INIT] Dropping privileges 2019-10-31 05:13:43,553 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:13:43,553 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:13:43,554 INFO [INIT] Marking all workers as offline 2019-10-31 05:13:43,595 INFO [INIT] LAVA master has started. 2019-10-31 05:13:43,596 INFO [INIT] Using protocol version 3 2019-10-31 05:13:51,285 DEBUG lava-logs => PING(20) 2019-10-31 05:14:02,374 DEBUG Dhanu => PING(20) 2019-10-31 05:14:02,374 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:14:03,597 INFO scheduling health checks: 2019-10-31 05:14:03,616 INFO scheduling jobs: 2019-10-31 05:14:03,618 DEBUG - qemu 2019-10-31 05:14:03,687 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:14:03,688 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:14:03,692 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:14:04,785 INFO [INIT] Dropping privileges 2019-10-31 05:14:04,785 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:14:04,785 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:14:04,787 INFO [INIT] Marking all workers as offline 2019-10-31 05:14:04,819 INFO [INIT] LAVA master has started. 2019-10-31 05:14:04,820 INFO [INIT] Using protocol version 3 2019-10-31 05:14:11,304 DEBUG lava-logs => PING(20) 2019-10-31 05:14:22,393 DEBUG Dhanu => PING(20) 2019-10-31 05:14:22,394 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:14:24,822 INFO scheduling health checks: 2019-10-31 05:14:24,852 INFO scheduling jobs: 2019-10-31 05:14:24,853 DEBUG - qemu 2019-10-31 05:14:24,918 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:14:24,918 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:14:24,923 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:14:26,035 INFO [INIT] Dropping privileges 2019-10-31 05:14:26,035 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:14:26,035 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:14:26,037 INFO [INIT] Marking all workers as offline 2019-10-31 05:14:26,076 INFO [INIT] LAVA master has started. 2019-10-31 05:14:26,076 INFO [INIT] Using protocol version 3 2019-10-31 05:14:31,323 DEBUG lava-logs => PING(20) 2019-10-31 05:14:42,414 DEBUG Dhanu => PING(20) 2019-10-31 05:14:42,415 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:14:46,079 INFO scheduling health checks: 2019-10-31 05:14:46,109 INFO scheduling jobs: 2019-10-31 05:14:46,111 DEBUG - qemu 2019-10-31 05:14:46,184 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:14:46,184 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:14:46,189 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:14:47,292 INFO [INIT] Dropping privileges 2019-10-31 05:14:47,292 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:14:47,292 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:14:47,294 INFO [INIT] Marking all workers as offline 2019-10-31 05:14:47,357 INFO [INIT] LAVA master has started. 2019-10-31 05:14:47,357 INFO [INIT] Using protocol version 3 2019-10-31 05:14:51,340 DEBUG lava-logs => PING(20) 2019-10-31 05:15:02,438 DEBUG Dhanu => PING(20) 2019-10-31 05:15:02,439 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:15:07,363 INFO scheduling health checks: 2019-10-31 05:15:07,389 INFO scheduling jobs: 2019-10-31 05:15:07,391 DEBUG - qemu 2019-10-31 05:15:07,464 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:15:07,465 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:15:07,473 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:15:08,617 INFO [INIT] Dropping privileges 2019-10-31 05:15:08,617 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:15:08,617 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:15:08,619 INFO [INIT] Marking all workers as offline 2019-10-31 05:15:08,646 INFO [INIT] LAVA master has started. 2019-10-31 05:15:08,646 INFO [INIT] Using protocol version 3 2019-10-31 05:15:11,355 DEBUG lava-logs => PING(20) 2019-10-31 05:15:22,460 DEBUG Dhanu => PING(20) 2019-10-31 05:15:22,460 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:15:28,651 INFO scheduling health checks: 2019-10-31 05:15:28,691 INFO scheduling jobs: 2019-10-31 05:15:28,694 DEBUG - qemu 2019-10-31 05:15:28,783 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:15:28,783 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:15:28,788 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:15:30,178 INFO [INIT] Dropping privileges 2019-10-31 05:15:30,178 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:15:30,178 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:15:30,180 INFO [INIT] Marking all workers as offline 2019-10-31 05:15:30,204 INFO [INIT] LAVA master has started. 2019-10-31 05:15:30,205 INFO [INIT] Using protocol version 3 2019-10-31 05:15:31,372 DEBUG lava-logs => PING(20) 2019-10-31 05:15:42,479 DEBUG Dhanu => PING(20) 2019-10-31 05:15:42,480 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:15:50,212 INFO scheduling health checks: 2019-10-31 05:15:50,249 INFO scheduling jobs: 2019-10-31 05:15:50,253 DEBUG - qemu 2019-10-31 05:15:50,327 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:15:50,328 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:15:50,332 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:15:51,555 INFO [INIT] Dropping privileges 2019-10-31 05:15:51,555 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:15:51,555 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:15:51,557 INFO [INIT] Marking all workers as offline 2019-10-31 05:15:51,630 INFO [INIT] LAVA master has started. 2019-10-31 05:15:51,630 INFO [INIT] Using protocol version 3 2019-10-31 05:15:51,731 DEBUG lava-logs => PING(20) 2019-10-31 05:16:02,496 DEBUG Dhanu => PING(20) 2019-10-31 05:16:02,497 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:16:11,634 INFO scheduling health checks: 2019-10-31 05:16:11,673 INFO scheduling jobs: 2019-10-31 05:16:11,677 DEBUG - qemu 2019-10-31 05:16:11,752 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:16:11,752 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:16:11,757 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:16:12,840 INFO [INIT] Dropping privileges 2019-10-31 05:16:12,841 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:16:12,841 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:16:12,842 INFO [INIT] Marking all workers as offline 2019-10-31 05:16:12,874 INFO [INIT] LAVA master has started. 2019-10-31 05:16:12,875 INFO [INIT] Using protocol version 3 2019-10-31 05:16:22,520 DEBUG Dhanu => PING(20) 2019-10-31 05:16:22,520 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:16:31,762 DEBUG lava-logs => PING(20) 2019-10-31 05:16:32,877 INFO scheduling health checks: 2019-10-31 05:16:32,897 INFO scheduling jobs: 2019-10-31 05:16:32,899 DEBUG - qemu 2019-10-31 05:16:32,965 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:16:32,965 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:16:32,969 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:16:34,292 INFO [INIT] Dropping privileges 2019-10-31 05:16:34,293 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:16:34,293 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:16:34,294 INFO [INIT] Marking all workers as offline 2019-10-31 05:16:34,321 INFO [INIT] LAVA master has started. 2019-10-31 05:16:34,322 INFO [INIT] Using protocol version 3 2019-10-31 05:16:42,538 DEBUG Dhanu => PING(20) 2019-10-31 05:16:42,538 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:16:51,776 DEBUG lava-logs => PING(20) 2019-10-31 05:16:54,323 INFO scheduling health checks: 2019-10-31 05:16:54,344 INFO scheduling jobs: 2019-10-31 05:16:54,346 DEBUG - qemu 2019-10-31 05:16:54,417 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:16:54,417 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:16:54,422 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:16:55,539 INFO [INIT] Dropping privileges 2019-10-31 05:16:55,539 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:16:55,540 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:16:55,541 INFO [INIT] Marking all workers as offline 2019-10-31 05:16:55,580 INFO [INIT] LAVA master has started. 2019-10-31 05:16:55,580 INFO [INIT] Using protocol version 3 2019-10-31 05:17:02,559 DEBUG Dhanu => PING(20) 2019-10-31 05:17:02,560 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:17:11,795 DEBUG lava-logs => PING(20) 2019-10-31 05:17:15,584 INFO scheduling health checks: 2019-10-31 05:17:15,613 INFO scheduling jobs: 2019-10-31 05:17:15,615 DEBUG - qemu 2019-10-31 05:17:15,687 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:17:15,688 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:17:15,692 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:17:16,781 INFO [INIT] Dropping privileges 2019-10-31 05:17:16,782 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:17:16,782 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:17:16,783 INFO [INIT] Marking all workers as offline 2019-10-31 05:17:16,815 INFO [INIT] LAVA master has started. 2019-10-31 05:17:16,815 INFO [INIT] Using protocol version 3 2019-10-31 05:17:22,580 DEBUG Dhanu => PING(20) 2019-10-31 05:17:22,580 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:17:31,807 DEBUG lava-logs => PING(20) 2019-10-31 05:17:36,821 INFO scheduling health checks: 2019-10-31 05:17:36,871 INFO scheduling jobs: 2019-10-31 05:17:36,874 DEBUG - qemu 2019-10-31 05:17:36,951 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:17:36,952 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:17:36,961 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:17:38,040 INFO [INIT] Dropping privileges 2019-10-31 05:17:38,041 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:17:38,041 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:17:38,042 INFO [INIT] Marking all workers as offline 2019-10-31 05:17:38,073 INFO [INIT] LAVA master has started. 2019-10-31 05:17:38,073 INFO [INIT] Using protocol version 3 2019-10-31 05:17:42,600 DEBUG Dhanu => PING(20) 2019-10-31 05:17:42,601 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:17:51,824 DEBUG lava-logs => PING(20) 2019-10-31 05:17:58,080 INFO scheduling health checks: 2019-10-31 05:17:58,117 INFO scheduling jobs: 2019-10-31 05:17:58,119 DEBUG - qemu 2019-10-31 05:17:58,188 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:17:58,188 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:17:58,198 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:17:59,287 INFO [INIT] Dropping privileges 2019-10-31 05:17:59,287 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:17:59,287 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:17:59,289 INFO [INIT] Marking all workers as offline 2019-10-31 05:17:59,320 INFO [INIT] LAVA master has started. 2019-10-31 05:17:59,320 INFO [INIT] Using protocol version 3 2019-10-31 05:18:02,622 DEBUG Dhanu => PING(20) 2019-10-31 05:18:02,622 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:18:11,839 DEBUG lava-logs => PING(20) 2019-10-31 05:18:19,328 INFO scheduling health checks: 2019-10-31 05:18:19,364 INFO scheduling jobs: 2019-10-31 05:18:19,367 DEBUG - qemu 2019-10-31 05:18:19,442 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:18:19,442 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:18:19,446 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:18:20,533 INFO [INIT] Dropping privileges 2019-10-31 05:18:20,533 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:18:20,533 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:18:20,534 INFO [INIT] Marking all workers as offline 2019-10-31 05:18:20,565 INFO [INIT] LAVA master has started. 2019-10-31 05:18:20,565 INFO [INIT] Using protocol version 3 2019-10-31 05:18:22,644 DEBUG Dhanu => PING(20) 2019-10-31 05:18:22,645 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:18:31,853 DEBUG lava-logs => PING(20) 2019-10-31 05:18:40,574 INFO scheduling health checks: 2019-10-31 05:18:40,610 INFO scheduling jobs: 2019-10-31 05:18:40,612 DEBUG - qemu 2019-10-31 05:18:40,679 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:18:40,679 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:18:40,683 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:18:41,791 INFO [INIT] Dropping privileges 2019-10-31 05:18:41,791 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:18:41,791 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:18:41,793 INFO [INIT] Marking all workers as offline 2019-10-31 05:18:41,835 INFO [INIT] LAVA master has started. 2019-10-31 05:18:41,835 INFO [INIT] Using protocol version 3 2019-10-31 05:18:42,664 DEBUG Dhanu => PING(20) 2019-10-31 05:18:42,665 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:18:51,866 DEBUG lava-logs => PING(20) 2019-10-31 05:19:01,846 INFO scheduling health checks: 2019-10-31 05:19:01,879 INFO scheduling jobs: 2019-10-31 05:19:01,881 DEBUG - qemu 2019-10-31 05:19:01,953 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:19:01,953 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:19:01,958 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:19:03,049 INFO [INIT] Dropping privileges 2019-10-31 05:19:03,050 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:19:03,050 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:19:03,051 INFO [INIT] Marking all workers as offline 2019-10-31 05:19:03,104 INFO [INIT] LAVA master has started. 2019-10-31 05:19:03,104 INFO [INIT] Using protocol version 3 2019-10-31 05:19:03,255 DEBUG Dhanu => PING(20) 2019-10-31 05:19:03,256 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:19:11,885 DEBUG lava-logs => PING(20) 2019-10-31 05:19:23,115 INFO scheduling health checks: 2019-10-31 05:19:23,170 INFO scheduling jobs: 2019-10-31 05:19:23,173 DEBUG - qemu 2019-10-31 05:19:23,248 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:19:23,249 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:19:23,253 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:19:24,551 INFO [INIT] Dropping privileges 2019-10-31 05:19:24,552 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:19:24,552 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:19:24,553 INFO [INIT] Marking all workers as offline 2019-10-31 05:19:24,606 INFO [INIT] LAVA master has started. 2019-10-31 05:19:24,606 INFO [INIT] Using protocol version 3 2019-10-31 05:19:24,666 DEBUG Dhanu => PING(20) 2019-10-31 05:19:24,667 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:19:31,906 DEBUG lava-logs => PING(20) 2019-10-31 05:19:44,619 INFO scheduling health checks: 2019-10-31 05:19:44,638 INFO scheduling jobs: 2019-10-31 05:19:44,640 DEBUG - qemu 2019-10-31 05:19:44,709 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:19:44,709 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:19:44,714 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:19:45,851 INFO [INIT] Dropping privileges 2019-10-31 05:19:45,852 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:19:45,852 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:19:45,853 INFO [INIT] Marking all workers as offline 2019-10-31 05:19:45,898 INFO [INIT] LAVA master has started. 2019-10-31 05:19:45,898 INFO [INIT] Using protocol version 3 2019-10-31 05:19:51,919 DEBUG lava-logs => PING(20) 2019-10-31 05:20:04,700 DEBUG Dhanu => PING(20) 2019-10-31 05:20:04,701 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:20:05,899 INFO scheduling health checks: 2019-10-31 05:20:05,940 INFO scheduling jobs: 2019-10-31 05:20:05,942 DEBUG - qemu 2019-10-31 05:20:06,013 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:20:06,013 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:20:06,018 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:20:07,337 INFO [INIT] Dropping privileges 2019-10-31 05:20:07,337 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:20:07,337 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:20:07,339 INFO [INIT] Marking all workers as offline 2019-10-31 05:20:07,365 INFO [INIT] LAVA master has started. 2019-10-31 05:20:07,365 INFO [INIT] Using protocol version 3 2019-10-31 05:20:11,932 DEBUG lava-logs => PING(20) 2019-10-31 05:20:24,716 DEBUG Dhanu => PING(20) 2019-10-31 05:20:24,717 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:20:27,367 INFO scheduling health checks: 2019-10-31 05:20:27,409 INFO scheduling jobs: 2019-10-31 05:20:27,412 DEBUG - qemu 2019-10-31 05:20:27,490 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:20:27,490 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:20:27,495 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:20:28,855 INFO [INIT] Dropping privileges 2019-10-31 05:20:28,856 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:20:28,856 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:20:28,857 INFO [INIT] Marking all workers as offline 2019-10-31 05:20:28,889 INFO [INIT] LAVA master has started. 2019-10-31 05:20:28,889 INFO [INIT] Using protocol version 3 2019-10-31 05:20:31,948 DEBUG lava-logs => PING(20) 2019-10-31 05:20:44,732 DEBUG Dhanu => PING(20) 2019-10-31 05:20:44,733 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:20:48,891 INFO scheduling health checks: 2019-10-31 05:20:48,933 INFO scheduling jobs: 2019-10-31 05:20:48,937 DEBUG - qemu 2019-10-31 05:20:49,013 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:20:49,013 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:20:49,018 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:20:50,324 INFO [INIT] Dropping privileges 2019-10-31 05:20:50,324 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:20:50,324 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:20:50,326 INFO [INIT] Marking all workers as offline 2019-10-31 05:20:50,358 INFO [INIT] LAVA master has started. 2019-10-31 05:20:50,358 INFO [INIT] Using protocol version 3 2019-10-31 05:20:51,962 DEBUG lava-logs => PING(20) 2019-10-31 05:21:04,748 DEBUG Dhanu => PING(20) 2019-10-31 05:21:04,749 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:21:10,363 INFO scheduling health checks: 2019-10-31 05:21:10,402 INFO scheduling jobs: 2019-10-31 05:21:10,404 DEBUG - qemu 2019-10-31 05:21:10,473 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:21:10,474 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:21:10,478 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:21:11,825 INFO [INIT] Dropping privileges 2019-10-31 05:21:11,825 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:21:11,825 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:21:11,827 INFO [INIT] Marking all workers as offline 2019-10-31 05:21:11,871 INFO [INIT] LAVA master has started. 2019-10-31 05:21:11,872 INFO [INIT] Using protocol version 3 2019-10-31 05:21:11,975 DEBUG lava-logs => PING(20) 2019-10-31 05:21:24,764 DEBUG Dhanu => PING(20) 2019-10-31 05:21:24,765 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:21:31,879 INFO scheduling health checks: 2019-10-31 05:21:31,915 INFO scheduling jobs: 2019-10-31 05:21:31,918 DEBUG - qemu 2019-10-31 05:21:31,993 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:21:31,993 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:21:31,998 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:21:33,384 INFO [INIT] Dropping privileges 2019-10-31 05:21:33,385 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:21:33,385 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:21:33,386 INFO [INIT] Marking all workers as offline 2019-10-31 05:21:33,430 INFO [INIT] LAVA master has started. 2019-10-31 05:21:33,430 INFO [INIT] Using protocol version 3 2019-10-31 05:21:44,777 DEBUG Dhanu => PING(20) 2019-10-31 05:21:44,777 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:21:52,008 DEBUG lava-logs => PING(20) 2019-10-31 05:21:53,431 INFO scheduling health checks: 2019-10-31 05:21:53,473 INFO scheduling jobs: 2019-10-31 05:21:53,476 DEBUG - qemu 2019-10-31 05:21:53,554 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:21:53,554 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:21:53,559 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:21:54,815 INFO [INIT] Dropping privileges 2019-10-31 05:21:54,815 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:21:54,815 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:21:54,817 INFO [INIT] Marking all workers as offline 2019-10-31 05:21:54,853 INFO [INIT] LAVA master has started. 2019-10-31 05:21:54,853 INFO [INIT] Using protocol version 3 2019-10-31 05:22:04,796 DEBUG Dhanu => PING(20) 2019-10-31 05:22:04,797 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:22:12,024 DEBUG lava-logs => PING(20) 2019-10-31 05:22:14,855 INFO scheduling health checks: 2019-10-31 05:22:14,886 INFO scheduling jobs: 2019-10-31 05:22:14,888 DEBUG - qemu 2019-10-31 05:22:14,959 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:22:14,959 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:22:14,964 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:22:16,076 INFO [INIT] Dropping privileges 2019-10-31 05:22:16,076 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:22:16,076 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:22:16,078 INFO [INIT] Marking all workers as offline 2019-10-31 05:22:16,111 INFO [INIT] LAVA master has started. 2019-10-31 05:22:16,111 INFO [INIT] Using protocol version 3 2019-10-31 05:22:24,812 DEBUG Dhanu => PING(20) 2019-10-31 05:22:24,813 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:22:32,040 DEBUG lava-logs => PING(20) 2019-10-31 05:22:36,115 INFO scheduling health checks: 2019-10-31 05:22:36,144 INFO scheduling jobs: 2019-10-31 05:22:36,146 DEBUG - qemu 2019-10-31 05:22:36,223 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:22:36,223 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:22:36,228 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:22:37,596 INFO [INIT] Dropping privileges 2019-10-31 05:22:37,597 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:22:37,597 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:22:37,598 INFO [INIT] Marking all workers as offline 2019-10-31 05:22:37,635 INFO [INIT] LAVA master has started. 2019-10-31 05:22:37,636 INFO [INIT] Using protocol version 3 2019-10-31 05:22:44,828 DEBUG Dhanu => PING(20) 2019-10-31 05:22:44,829 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:22:52,052 DEBUG lava-logs => PING(20) 2019-10-31 05:22:57,639 INFO scheduling health checks: 2019-10-31 05:22:57,679 INFO scheduling jobs: 2019-10-31 05:22:57,681 DEBUG - qemu 2019-10-31 05:22:57,761 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:22:57,761 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:22:57,799 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:22:59,197 INFO [INIT] Dropping privileges 2019-10-31 05:22:59,197 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:22:59,197 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:22:59,200 INFO [INIT] Marking all workers as offline 2019-10-31 05:22:59,256 INFO [INIT] LAVA master has started. 2019-10-31 05:22:59,257 INFO [INIT] Using protocol version 3 2019-10-31 05:23:04,840 DEBUG Dhanu => PING(20) 2019-10-31 05:23:04,841 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:23:12,067 DEBUG lava-logs => PING(20) 2019-10-31 05:23:19,263 INFO scheduling health checks: 2019-10-31 05:23:19,303 INFO scheduling jobs: 2019-10-31 05:23:19,305 DEBUG - qemu 2019-10-31 05:23:19,380 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:23:19,381 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:23:19,390 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:23:20,573 INFO [INIT] Dropping privileges 2019-10-31 05:23:20,574 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:23:20,574 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:23:20,575 INFO [INIT] Marking all workers as offline 2019-10-31 05:23:20,617 INFO [INIT] LAVA master has started. 2019-10-31 05:23:20,617 INFO [INIT] Using protocol version 3 2019-10-31 05:23:24,857 DEBUG Dhanu => PING(20) 2019-10-31 05:23:24,858 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:23:32,080 DEBUG lava-logs => PING(20) 2019-10-31 05:23:40,625 INFO scheduling health checks: 2019-10-31 05:23:40,661 INFO scheduling jobs: 2019-10-31 05:23:40,663 DEBUG - qemu 2019-10-31 05:23:40,740 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:23:40,740 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:23:40,745 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:23:42,072 INFO [INIT] Dropping privileges 2019-10-31 05:23:42,073 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:23:42,073 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:23:42,074 INFO [INIT] Marking all workers as offline 2019-10-31 05:23:42,108 INFO [INIT] LAVA master has started. 2019-10-31 05:23:42,109 INFO [INIT] Using protocol version 3 2019-10-31 05:23:44,872 DEBUG Dhanu => PING(20) 2019-10-31 05:23:44,873 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:23:52,096 DEBUG lava-logs => PING(20) 2019-10-31 05:24:02,119 INFO scheduling health checks: 2019-10-31 05:24:02,163 INFO scheduling jobs: 2019-10-31 05:24:02,166 DEBUG - qemu 2019-10-31 05:24:02,242 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:24:02,242 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:24:02,247 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:24:03,824 INFO [INIT] Dropping privileges 2019-10-31 05:24:03,874 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:24:03,874 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:24:03,881 INFO [INIT] Marking all workers as offline 2019-10-31 05:24:03,962 INFO [INIT] LAVA master has started. 2019-10-31 05:24:03,962 INFO [INIT] Using protocol version 3 2019-10-31 05:24:04,885 DEBUG Dhanu => PING(20) 2019-10-31 05:24:04,886 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:24:12,116 DEBUG lava-logs => PING(20) 2019-10-31 05:24:23,973 INFO scheduling health checks: 2019-10-31 05:24:23,998 INFO scheduling jobs: 2019-10-31 05:24:24,000 DEBUG - qemu 2019-10-31 05:24:24,078 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:24:24,079 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:24:24,126 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:24:25,334 INFO [INIT] Dropping privileges 2019-10-31 05:24:25,335 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:24:25,335 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:24:25,336 INFO [INIT] Marking all workers as offline 2019-10-31 05:24:25,368 INFO [INIT] LAVA master has started. 2019-10-31 05:24:25,368 INFO [INIT] Using protocol version 3 2019-10-31 05:24:25,488 DEBUG Dhanu => PING(20) 2019-10-31 05:24:25,489 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:24:32,132 DEBUG lava-logs => PING(20) 2019-10-31 05:24:45,379 INFO scheduling health checks: 2019-10-31 05:24:45,412 INFO scheduling jobs: 2019-10-31 05:24:45,413 DEBUG - qemu 2019-10-31 05:24:45,481 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:24:45,481 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:24:45,486 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:24:46,830 INFO [INIT] Dropping privileges 2019-10-31 05:24:46,830 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:24:46,830 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:24:46,832 INFO [INIT] Marking all workers as offline 2019-10-31 05:24:46,858 INFO [INIT] LAVA master has started. 2019-10-31 05:24:46,858 INFO [INIT] Using protocol version 3 2019-10-31 05:24:46,878 DEBUG Dhanu => PING(20) 2019-10-31 05:24:46,879 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:24:52,148 DEBUG lava-logs => PING(20) 2019-10-31 05:25:06,871 INFO scheduling health checks: 2019-10-31 05:25:06,914 INFO scheduling jobs: 2019-10-31 05:25:06,916 DEBUG - qemu 2019-10-31 05:25:07,002 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:25:07,003 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:25:07,054 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:25:08,532 INFO [INIT] Dropping privileges 2019-10-31 05:25:08,533 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:25:08,533 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:25:08,535 INFO [INIT] Marking all workers as offline 2019-10-31 05:25:08,591 INFO [INIT] LAVA master has started. 2019-10-31 05:25:08,591 INFO [INIT] Using protocol version 3 2019-10-31 05:25:12,160 DEBUG lava-logs => PING(20) 2019-10-31 05:25:26,929 DEBUG Dhanu => PING(20) 2019-10-31 05:25:26,930 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:25:28,594 INFO scheduling health checks: 2019-10-31 05:25:28,627 INFO scheduling jobs: 2019-10-31 05:25:28,629 DEBUG - qemu 2019-10-31 05:25:28,707 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:25:28,708 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:25:28,756 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:25:30,159 INFO [INIT] Dropping privileges 2019-10-31 05:25:30,160 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:25:30,160 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:25:30,161 INFO [INIT] Marking all workers as offline 2019-10-31 05:25:30,207 INFO [INIT] LAVA master has started. 2019-10-31 05:25:30,208 INFO [INIT] Using protocol version 3 2019-10-31 05:25:32,175 DEBUG lava-logs => PING(20) 2019-10-31 05:25:46,945 DEBUG Dhanu => PING(20) 2019-10-31 05:25:46,945 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:25:50,211 INFO scheduling health checks: 2019-10-31 05:25:50,274 INFO scheduling jobs: 2019-10-31 05:25:50,277 DEBUG - qemu 2019-10-31 05:25:50,355 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:25:50,355 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:25:50,360 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:25:51,593 INFO [INIT] Dropping privileges 2019-10-31 05:25:51,594 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:25:51,594 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:25:51,595 INFO [INIT] Marking all workers as offline 2019-10-31 05:25:51,631 INFO [INIT] LAVA master has started. 2019-10-31 05:25:51,631 INFO [INIT] Using protocol version 3 2019-10-31 05:25:52,184 DEBUG lava-logs => PING(20) 2019-10-31 05:26:06,960 DEBUG Dhanu => PING(20) 2019-10-31 05:26:06,961 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:26:11,635 INFO scheduling health checks: 2019-10-31 05:26:11,664 INFO scheduling jobs: 2019-10-31 05:26:11,666 DEBUG - qemu 2019-10-31 05:26:11,735 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:26:11,735 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:26:11,740 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:26:12,873 INFO [INIT] Dropping privileges 2019-10-31 05:26:12,874 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:26:12,874 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:26:12,876 INFO [INIT] Marking all workers as offline 2019-10-31 05:26:12,899 INFO [INIT] LAVA master has started. 2019-10-31 05:26:12,900 INFO [INIT] Using protocol version 3 2019-10-31 05:26:12,953 DEBUG lava-logs => PING(20) 2019-10-31 05:26:27,024 DEBUG Dhanu => PING(20) 2019-10-31 05:26:27,024 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:26:32,903 INFO scheduling health checks: 2019-10-31 05:26:32,944 INFO scheduling jobs: 2019-10-31 05:26:32,947 DEBUG - qemu 2019-10-31 05:26:33,026 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:26:33,026 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:26:33,107 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:26:34,525 INFO [INIT] Dropping privileges 2019-10-31 05:26:34,543 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:26:34,543 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:26:34,546 INFO [INIT] Marking all workers as offline 2019-10-31 05:26:34,600 INFO [INIT] LAVA master has started. 2019-10-31 05:26:34,600 INFO [INIT] Using protocol version 3 2019-10-31 05:26:47,036 DEBUG Dhanu => PING(20) 2019-10-31 05:26:47,037 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:26:52,988 DEBUG lava-logs => PING(20) 2019-10-31 05:26:54,603 INFO scheduling health checks: 2019-10-31 05:26:54,624 INFO scheduling jobs: 2019-10-31 05:26:54,625 DEBUG - qemu 2019-10-31 05:26:54,694 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:26:54,694 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:26:54,699 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:26:55,837 INFO [INIT] Dropping privileges 2019-10-31 05:26:55,838 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:26:55,838 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:26:55,839 INFO [INIT] Marking all workers as offline 2019-10-31 05:26:55,871 INFO [INIT] LAVA master has started. 2019-10-31 05:26:55,872 INFO [INIT] Using protocol version 3 2019-10-31 05:27:07,048 DEBUG Dhanu => PING(20) 2019-10-31 05:27:07,049 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:27:13,004 DEBUG lava-logs => PING(20) 2019-10-31 05:27:15,875 INFO scheduling health checks: 2019-10-31 05:27:15,904 INFO scheduling jobs: 2019-10-31 05:27:15,907 DEBUG - qemu 2019-10-31 05:27:15,984 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:27:15,984 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:27:15,989 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:27:17,316 INFO [INIT] Dropping privileges 2019-10-31 05:27:17,317 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:27:17,317 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:27:17,318 INFO [INIT] Marking all workers as offline 2019-10-31 05:27:17,397 INFO [INIT] LAVA master has started. 2019-10-31 05:27:17,397 INFO [INIT] Using protocol version 3 2019-10-31 05:27:27,064 DEBUG Dhanu => PING(20) 2019-10-31 05:27:27,065 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:27:33,016 DEBUG lava-logs => PING(20) 2019-10-31 05:27:37,399 INFO scheduling health checks: 2019-10-31 05:27:37,473 INFO scheduling jobs: 2019-10-31 05:27:37,475 DEBUG - qemu 2019-10-31 05:27:37,577 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:27:37,577 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:27:37,614 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:27:39,380 INFO [INIT] Dropping privileges 2019-10-31 05:27:39,394 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:27:39,394 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:27:39,397 INFO [INIT] Marking all workers as offline 2019-10-31 05:27:39,470 INFO [INIT] LAVA master has started. 2019-10-31 05:27:39,470 INFO [INIT] Using protocol version 3 2019-10-31 05:27:47,080 DEBUG Dhanu => PING(20) 2019-10-31 05:27:47,081 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:27:53,028 DEBUG lava-logs => PING(20) 2019-10-31 05:27:59,475 INFO scheduling health checks: 2019-10-31 05:27:59,519 INFO scheduling jobs: 2019-10-31 05:27:59,523 DEBUG - qemu 2019-10-31 05:27:59,606 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:27:59,606 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:27:59,610 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:28:00,835 INFO [INIT] Dropping privileges 2019-10-31 05:28:00,835 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:28:00,835 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:28:00,837 INFO [INIT] Marking all workers as offline 2019-10-31 05:28:00,877 INFO [INIT] LAVA master has started. 2019-10-31 05:28:00,878 INFO [INIT] Using protocol version 3 2019-10-31 05:28:07,093 DEBUG Dhanu => PING(20) 2019-10-31 05:28:07,093 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:28:13,040 DEBUG lava-logs => PING(20) 2019-10-31 05:28:20,883 INFO scheduling health checks: 2019-10-31 05:28:20,921 INFO scheduling jobs: 2019-10-31 05:28:20,924 DEBUG - qemu 2019-10-31 05:28:21,014 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:28:21,014 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:28:21,023 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:28:22,456 INFO [INIT] Dropping privileges 2019-10-31 05:28:22,456 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:28:22,456 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:28:22,458 INFO [INIT] Marking all workers as offline 2019-10-31 05:28:22,490 INFO [INIT] LAVA master has started. 2019-10-31 05:28:22,490 INFO [INIT] Using protocol version 3 2019-10-31 05:28:27,108 DEBUG Dhanu => PING(20) 2019-10-31 05:28:27,110 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:28:33,052 DEBUG lava-logs => PING(20) 2019-10-31 05:28:42,499 INFO scheduling health checks: 2019-10-31 05:28:42,535 INFO scheduling jobs: 2019-10-31 05:28:42,538 DEBUG - qemu 2019-10-31 05:28:42,628 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:28:42,628 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:28:42,664 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:28:43,959 INFO [INIT] Dropping privileges 2019-10-31 05:28:43,960 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:28:43,960 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:28:43,962 INFO [INIT] Marking all workers as offline 2019-10-31 05:28:44,004 INFO [INIT] LAVA master has started. 2019-10-31 05:28:44,004 INFO [INIT] Using protocol version 3 2019-10-31 05:28:47,128 DEBUG Dhanu => PING(20) 2019-10-31 05:28:47,129 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:28:53,072 DEBUG lava-logs => PING(20) 2019-10-31 05:29:04,011 INFO scheduling health checks: 2019-10-31 05:29:04,048 INFO scheduling jobs: 2019-10-31 05:29:04,050 DEBUG - qemu 2019-10-31 05:29:04,125 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:29:04,125 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:29:04,160 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:29:05,624 INFO [INIT] Dropping privileges 2019-10-31 05:29:05,625 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:29:05,626 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:29:05,630 INFO [INIT] Marking all workers as offline 2019-10-31 05:29:05,662 INFO [INIT] LAVA master has started. 2019-10-31 05:29:05,662 INFO [INIT] Using protocol version 3 2019-10-31 05:29:07,142 DEBUG Dhanu => PING(20) 2019-10-31 05:29:07,142 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:29:13,090 DEBUG lava-logs => PING(20) 2019-10-31 05:29:25,671 INFO scheduling health checks: 2019-10-31 05:29:25,705 INFO scheduling jobs: 2019-10-31 05:29:25,707 DEBUG - qemu 2019-10-31 05:29:25,780 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:29:25,780 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:29:25,785 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:29:27,087 INFO [INIT] Dropping privileges 2019-10-31 05:29:27,088 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:29:27,088 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:29:27,089 INFO [INIT] Marking all workers as offline 2019-10-31 05:29:27,130 INFO [INIT] LAVA master has started. 2019-10-31 05:29:27,130 INFO [INIT] Using protocol version 3 2019-10-31 05:29:27,156 DEBUG Dhanu => PING(20) 2019-10-31 05:29:27,157 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:29:33,104 DEBUG lava-logs => PING(20) 2019-10-31 05:29:47,143 INFO scheduling health checks: 2019-10-31 05:29:47,174 INFO scheduling jobs: 2019-10-31 05:29:47,176 DEBUG - qemu 2019-10-31 05:29:47,260 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:29:47,260 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:29:47,268 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:29:48,578 INFO [INIT] Dropping privileges 2019-10-31 05:29:48,578 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:29:48,578 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:29:48,580 INFO [INIT] Marking all workers as offline 2019-10-31 05:29:48,621 INFO [INIT] LAVA master has started. 2019-10-31 05:29:48,622 INFO [INIT] Using protocol version 3 2019-10-31 05:29:53,121 DEBUG lava-logs => PING(20) 2019-10-31 05:30:07,211 DEBUG Dhanu => PING(20) 2019-10-31 05:30:07,212 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:30:08,623 INFO scheduling health checks: 2019-10-31 05:30:08,654 INFO scheduling jobs: 2019-10-31 05:30:08,656 DEBUG - qemu 2019-10-31 05:30:08,729 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:30:08,729 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:30:08,765 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:30:10,260 INFO [INIT] Dropping privileges 2019-10-31 05:30:10,260 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:30:10,260 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:30:10,262 INFO [INIT] Marking all workers as offline 2019-10-31 05:30:10,353 INFO [INIT] LAVA master has started. 2019-10-31 05:30:10,354 INFO [INIT] Using protocol version 3 2019-10-31 05:30:13,135 DEBUG lava-logs => PING(20) 2019-10-31 05:30:27,229 DEBUG Dhanu => PING(20) 2019-10-31 05:30:27,229 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:30:30,355 INFO scheduling health checks: 2019-10-31 05:30:30,389 INFO scheduling jobs: 2019-10-31 05:30:30,391 DEBUG - qemu 2019-10-31 05:30:30,479 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:30:30,479 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:30:30,484 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:30:31,961 INFO [INIT] Dropping privileges 2019-10-31 05:30:31,961 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:30:31,961 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:30:31,963 INFO [INIT] Marking all workers as offline 2019-10-31 05:30:32,015 INFO [INIT] LAVA master has started. 2019-10-31 05:30:32,016 INFO [INIT] Using protocol version 3 2019-10-31 05:30:33,145 DEBUG lava-logs => PING(20) 2019-10-31 05:30:47,244 DEBUG Dhanu => PING(20) 2019-10-31 05:30:47,245 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:30:52,019 INFO scheduling health checks: 2019-10-31 05:30:52,058 INFO scheduling jobs: 2019-10-31 05:30:52,062 DEBUG - qemu 2019-10-31 05:30:52,140 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:30:52,141 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:30:52,145 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:30:53,450 INFO [INIT] Dropping privileges 2019-10-31 05:30:53,450 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:30:53,450 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:30:53,452 INFO [INIT] Marking all workers as offline 2019-10-31 05:30:53,495 INFO [INIT] LAVA master has started. 2019-10-31 05:30:53,496 INFO [INIT] Using protocol version 3 2019-10-31 05:30:53,512 DEBUG lava-logs => PING(20) 2019-10-31 05:31:07,273 DEBUG Dhanu => PING(20) 2019-10-31 05:31:07,282 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:31:13,499 INFO scheduling health checks: 2019-10-31 05:31:13,549 INFO scheduling jobs: 2019-10-31 05:31:13,551 DEBUG - qemu 2019-10-31 05:31:13,620 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:31:13,620 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:31:13,667 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:31:15,393 INFO [INIT] Dropping privileges 2019-10-31 05:31:15,393 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:31:15,393 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:31:15,395 INFO [INIT] Marking all workers as offline 2019-10-31 05:31:15,447 INFO [INIT] LAVA master has started. 2019-10-31 05:31:15,447 INFO [INIT] Using protocol version 3 2019-10-31 05:31:27,296 DEBUG Dhanu => PING(20) 2019-10-31 05:31:27,297 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:31:33,548 DEBUG lava-logs => PING(20) 2019-10-31 05:31:35,450 INFO scheduling health checks: 2019-10-31 05:31:35,485 INFO scheduling jobs: 2019-10-31 05:31:35,486 DEBUG - qemu 2019-10-31 05:31:35,555 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:31:35,555 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:31:35,560 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:31:36,819 INFO [INIT] Dropping privileges 2019-10-31 05:31:36,819 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:31:36,819 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:31:36,838 INFO [INIT] Marking all workers as offline 2019-10-31 05:31:36,876 INFO [INIT] LAVA master has started. 2019-10-31 05:31:36,877 INFO [INIT] Using protocol version 3 2019-10-31 05:31:47,308 DEBUG Dhanu => PING(20) 2019-10-31 05:31:47,309 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:31:53,564 DEBUG lava-logs => PING(20) 2019-10-31 05:31:56,879 INFO scheduling health checks: 2019-10-31 05:31:56,920 INFO scheduling jobs: 2019-10-31 05:31:56,923 DEBUG - qemu 2019-10-31 05:31:57,000 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:31:57,000 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:31:57,005 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:31:58,309 INFO [INIT] Dropping privileges 2019-10-31 05:31:58,310 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:31:58,310 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:31:58,311 INFO [INIT] Marking all workers as offline 2019-10-31 05:31:58,345 INFO [INIT] LAVA master has started. 2019-10-31 05:31:58,345 INFO [INIT] Using protocol version 3 2019-10-31 05:32:07,328 DEBUG Dhanu => PING(20) 2019-10-31 05:32:07,329 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:32:13,576 DEBUG lava-logs => PING(20) 2019-10-31 05:32:18,374 INFO scheduling health checks: 2019-10-31 05:32:18,467 INFO scheduling jobs: 2019-10-31 05:32:18,469 DEBUG - qemu 2019-10-31 05:32:18,558 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:32:18,558 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:32:18,596 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:32:20,320 INFO [INIT] Dropping privileges 2019-10-31 05:32:20,320 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:32:20,320 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:32:20,322 INFO [INIT] Marking all workers as offline 2019-10-31 05:32:20,376 INFO [INIT] LAVA master has started. 2019-10-31 05:32:20,376 INFO [INIT] Using protocol version 3 2019-10-31 05:32:27,358 DEBUG Dhanu => PING(20) 2019-10-31 05:32:27,359 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:32:33,591 DEBUG lava-logs => PING(20) 2019-10-31 05:32:40,379 INFO scheduling health checks: 2019-10-31 05:32:40,402 INFO scheduling jobs: 2019-10-31 05:32:40,403 DEBUG - qemu 2019-10-31 05:32:40,473 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:32:40,473 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:32:40,503 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:32:41,836 INFO [INIT] Dropping privileges 2019-10-31 05:32:41,836 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:32:41,837 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:32:41,838 INFO [INIT] Marking all workers as offline 2019-10-31 05:32:41,872 INFO [INIT] LAVA master has started. 2019-10-31 05:32:41,872 INFO [INIT] Using protocol version 3 2019-10-31 05:32:47,372 DEBUG Dhanu => PING(20) 2019-10-31 05:32:47,373 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:32:53,604 DEBUG lava-logs => PING(20) 2019-10-31 05:33:01,880 INFO scheduling health checks: 2019-10-31 05:33:01,915 INFO scheduling jobs: 2019-10-31 05:33:01,917 DEBUG - qemu 2019-10-31 05:33:01,994 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:33:01,995 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:33:01,999 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:33:03,329 INFO [INIT] Dropping privileges 2019-10-31 05:33:03,329 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:33:03,329 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:33:03,331 INFO [INIT] Marking all workers as offline 2019-10-31 05:33:03,374 INFO [INIT] LAVA master has started. 2019-10-31 05:33:03,374 INFO [INIT] Using protocol version 3 2019-10-31 05:33:07,392 DEBUG Dhanu => PING(20) 2019-10-31 05:33:07,393 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:33:13,616 DEBUG lava-logs => PING(20) 2019-10-31 05:33:23,383 INFO scheduling health checks: 2019-10-31 05:33:23,428 INFO scheduling jobs: 2019-10-31 05:33:23,431 DEBUG - qemu 2019-10-31 05:33:23,517 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:33:23,517 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:33:23,522 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:33:24,918 INFO [INIT] Dropping privileges 2019-10-31 05:33:24,918 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:33:24,918 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:33:24,920 INFO [INIT] Marking all workers as offline 2019-10-31 05:33:24,987 INFO [INIT] LAVA master has started. 2019-10-31 05:33:24,987 INFO [INIT] Using protocol version 3 2019-10-31 05:33:27,419 DEBUG Dhanu => PING(20) 2019-10-31 05:33:27,420 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:33:33,632 DEBUG lava-logs => PING(20) 2019-10-31 05:33:44,996 INFO scheduling health checks: 2019-10-31 05:33:45,030 INFO scheduling jobs: 2019-10-31 05:33:45,032 DEBUG - qemu 2019-10-31 05:33:45,119 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:33:45,119 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:33:45,159 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:33:46,904 INFO [INIT] Dropping privileges 2019-10-31 05:33:46,904 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:33:46,904 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:33:46,919 INFO [INIT] Marking all workers as offline 2019-10-31 05:33:46,973 INFO [INIT] LAVA master has started. 2019-10-31 05:33:46,973 INFO [INIT] Using protocol version 3 2019-10-31 05:33:47,433 DEBUG Dhanu => PING(20) 2019-10-31 05:33:47,434 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:33:53,648 DEBUG lava-logs => PING(20) 2019-10-31 05:34:06,983 INFO scheduling health checks: 2019-10-31 05:34:07,021 INFO scheduling jobs: 2019-10-31 05:34:07,024 DEBUG - qemu 2019-10-31 05:34:07,110 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:34:07,111 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:34:07,116 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:34:08,587 INFO [INIT] Dropping privileges 2019-10-31 05:34:08,588 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:34:08,588 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:34:08,589 INFO [INIT] Marking all workers as offline 2019-10-31 05:34:08,635 INFO [INIT] LAVA master has started. 2019-10-31 05:34:08,636 INFO [INIT] Using protocol version 3 2019-10-31 05:34:08,722 DEBUG Dhanu => PING(20) 2019-10-31 05:34:08,724 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:34:13,660 DEBUG lava-logs => PING(20) 2019-10-31 05:34:28,649 INFO scheduling health checks: 2019-10-31 05:34:28,679 INFO scheduling jobs: 2019-10-31 05:34:28,681 DEBUG - qemu 2019-10-31 05:34:28,768 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:34:28,768 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:34:28,775 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:34:30,144 INFO [INIT] Dropping privileges 2019-10-31 05:34:30,145 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:34:30,145 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:34:30,146 INFO [INIT] Marking all workers as offline 2019-10-31 05:34:30,170 INFO [INIT] LAVA master has started. 2019-10-31 05:34:30,170 INFO [INIT] Using protocol version 3 2019-10-31 05:34:33,672 DEBUG lava-logs => PING(20) 2019-10-31 05:34:48,794 DEBUG Dhanu => PING(20) 2019-10-31 05:34:48,848 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:34:50,171 INFO scheduling health checks: 2019-10-31 05:34:50,204 INFO scheduling jobs: 2019-10-31 05:34:50,206 DEBUG - qemu 2019-10-31 05:34:50,284 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:34:50,284 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:34:50,319 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:34:52,043 INFO [INIT] Dropping privileges 2019-10-31 05:34:52,043 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:34:52,043 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:34:52,058 INFO [INIT] Marking all workers as offline 2019-10-31 05:34:52,134 INFO [INIT] LAVA master has started. 2019-10-31 05:34:52,135 INFO [INIT] Using protocol version 3 2019-10-31 05:34:53,686 DEBUG lava-logs => PING(20) 2019-10-31 05:35:08,864 DEBUG Dhanu => PING(20) 2019-10-31 05:35:08,865 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:35:12,137 INFO scheduling health checks: 2019-10-31 05:35:12,161 INFO scheduling jobs: 2019-10-31 05:35:12,163 DEBUG - qemu 2019-10-31 05:35:12,241 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:35:12,242 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:35:12,247 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:35:13,587 INFO [INIT] Dropping privileges 2019-10-31 05:35:13,588 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:35:13,588 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:35:13,589 INFO [INIT] Marking all workers as offline 2019-10-31 05:35:13,678 INFO [INIT] LAVA master has started. 2019-10-31 05:35:13,678 INFO [INIT] Using protocol version 3 2019-10-31 05:35:13,774 DEBUG lava-logs => PING(20) 2019-10-31 05:35:28,880 DEBUG Dhanu => PING(20) 2019-10-31 05:35:28,881 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:35:33,683 INFO scheduling health checks: 2019-10-31 05:35:33,719 INFO scheduling jobs: 2019-10-31 05:35:33,720 DEBUG - qemu 2019-10-31 05:35:33,790 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:35:33,790 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:35:33,795 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:35:35,081 INFO [INIT] Dropping privileges 2019-10-31 05:35:35,081 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:35:35,081 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:35:35,083 INFO [INIT] Marking all workers as offline 2019-10-31 05:35:35,121 INFO [INIT] LAVA master has started. 2019-10-31 05:35:35,122 INFO [INIT] Using protocol version 3 2019-10-31 05:35:48,896 DEBUG Dhanu => PING(20) 2019-10-31 05:35:48,897 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:35:53,845 DEBUG lava-logs => PING(20) 2019-10-31 05:35:55,123 INFO scheduling health checks: 2019-10-31 05:35:55,176 INFO scheduling jobs: 2019-10-31 05:35:55,179 DEBUG - qemu 2019-10-31 05:35:55,253 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:35:55,253 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:35:55,304 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:35:56,857 INFO [INIT] Dropping privileges 2019-10-31 05:35:56,857 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:35:56,858 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:35:56,859 INFO [INIT] Marking all workers as offline 2019-10-31 05:35:56,942 INFO [INIT] LAVA master has started. 2019-10-31 05:35:56,943 INFO [INIT] Using protocol version 3 2019-10-31 05:36:08,913 DEBUG Dhanu => PING(20) 2019-10-31 05:36:08,915 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:36:13,860 DEBUG lava-logs => PING(20) 2019-10-31 05:36:16,947 INFO scheduling health checks: 2019-10-31 05:36:16,990 INFO scheduling jobs: 2019-10-31 05:36:16,993 DEBUG - qemu 2019-10-31 05:36:17,074 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:36:17,074 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:36:17,119 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:36:18,444 INFO [INIT] Dropping privileges 2019-10-31 05:36:18,444 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:36:18,444 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:36:18,446 INFO [INIT] Marking all workers as offline 2019-10-31 05:36:18,481 INFO [INIT] LAVA master has started. 2019-10-31 05:36:18,481 INFO [INIT] Using protocol version 3 2019-10-31 05:36:28,928 DEBUG Dhanu => PING(20) 2019-10-31 05:36:28,929 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:36:33,876 DEBUG lava-logs => PING(20) 2019-10-31 05:36:38,483 INFO scheduling health checks: 2019-10-31 05:36:38,514 INFO scheduling jobs: 2019-10-31 05:36:38,515 DEBUG - qemu 2019-10-31 05:36:38,591 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:36:38,592 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:36:38,600 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:36:39,850 INFO [INIT] Dropping privileges 2019-10-31 05:36:39,851 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:36:39,851 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:36:39,853 INFO [INIT] Marking all workers as offline 2019-10-31 05:36:39,894 INFO [INIT] LAVA master has started. 2019-10-31 05:36:39,895 INFO [INIT] Using protocol version 3 2019-10-31 05:36:44,181 INFO [POLL] Received a signal, leaving 2019-10-31 05:36:44,181 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:36:45,197 INFO [INIT] Dropping privileges 2019-10-31 05:36:45,197 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:36:45,198 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:36:45,199 INFO [INIT] Marking all workers as offline 2019-10-31 05:36:45,239 INFO [INIT] LAVA master has started. 2019-10-31 05:36:45,240 INFO [INIT] Using protocol version 3 2019-10-31 05:36:48,944 DEBUG Dhanu => PING(20) 2019-10-31 05:36:48,945 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:36:49,571 INFO Dhanu => HELLO 2019-10-31 05:36:49,571 WARNING Dispatcher has RESTARTED 2019-10-31 05:36:53,888 DEBUG lava-logs => PING(20) 2019-10-31 05:36:54,576 DEBUG Dhanu => PING(20) 2019-10-31 05:37:05,242 INFO scheduling health checks: 2019-10-31 05:37:05,272 INFO scheduling jobs: 2019-10-31 05:37:05,275 DEBUG - qemu 2019-10-31 05:37:05,351 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:37:05,351 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:37:05,356 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:37:06,598 INFO [INIT] Dropping privileges 2019-10-31 05:37:06,598 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:37:06,598 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:37:06,600 INFO [INIT] Marking all workers as offline 2019-10-31 05:37:06,641 INFO [INIT] LAVA master has started. 2019-10-31 05:37:06,641 INFO [INIT] Using protocol version 3 2019-10-31 05:37:14,665 DEBUG Dhanu => PING(20) 2019-10-31 05:37:14,665 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:37:22,811 DEBUG lava-logs => PING(20) 2019-10-31 05:37:22,829 DEBUG lava-logs => PING(20) 2019-10-31 05:37:26,645 INFO scheduling health checks: 2019-10-31 05:37:26,663 INFO scheduling jobs: 2019-10-31 05:37:26,665 DEBUG - qemu 2019-10-31 05:37:26,742 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:37:26,743 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:37:26,776 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:37:28,295 INFO [INIT] Dropping privileges 2019-10-31 05:37:28,296 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:37:28,296 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:37:28,299 INFO [INIT] Marking all workers as offline 2019-10-31 05:37:28,343 INFO [INIT] LAVA master has started. 2019-10-31 05:37:28,343 INFO [INIT] Using protocol version 3 2019-10-31 05:37:34,700 DEBUG Dhanu => PING(20) 2019-10-31 05:37:34,700 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:37:42,844 DEBUG lava-logs => PING(20) 2019-10-31 05:37:48,347 INFO scheduling health checks: 2019-10-31 05:37:48,454 INFO scheduling jobs: 2019-10-31 05:37:48,456 DEBUG - qemu 2019-10-31 05:37:48,549 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:37:48,549 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:37:48,554 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:37:49,871 INFO [INIT] Dropping privileges 2019-10-31 05:37:49,871 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:37:49,871 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:37:49,873 INFO [INIT] Marking all workers as offline 2019-10-31 05:37:49,923 INFO [INIT] LAVA master has started. 2019-10-31 05:37:49,923 INFO [INIT] Using protocol version 3 2019-10-31 05:37:54,716 DEBUG Dhanu => PING(20) 2019-10-31 05:37:54,717 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:38:02,856 DEBUG lava-logs => PING(20) 2019-10-31 05:38:09,927 INFO scheduling health checks: 2019-10-31 05:38:09,967 INFO scheduling jobs: 2019-10-31 05:38:09,970 DEBUG - qemu 2019-10-31 05:38:10,064 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:38:10,064 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:38:10,069 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:38:11,496 INFO [INIT] Dropping privileges 2019-10-31 05:38:11,496 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:38:11,496 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:38:11,500 INFO [INIT] Marking all workers as offline 2019-10-31 05:38:11,539 INFO [INIT] LAVA master has started. 2019-10-31 05:38:11,539 INFO [INIT] Using protocol version 3 2019-10-31 05:38:14,732 DEBUG Dhanu => PING(20) 2019-10-31 05:38:14,732 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:38:22,868 DEBUG lava-logs => PING(20) 2019-10-31 05:38:31,547 INFO scheduling health checks: 2019-10-31 05:38:31,592 INFO scheduling jobs: 2019-10-31 05:38:31,594 DEBUG - qemu 2019-10-31 05:38:31,678 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:38:31,678 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:38:31,685 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:38:33,106 INFO [INIT] Dropping privileges 2019-10-31 05:38:33,106 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:38:33,106 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:38:33,109 INFO [INIT] Marking all workers as offline 2019-10-31 05:38:33,162 INFO [INIT] LAVA master has started. 2019-10-31 05:38:33,162 INFO [INIT] Using protocol version 3 2019-10-31 05:38:34,746 DEBUG Dhanu => PING(20) 2019-10-31 05:38:34,746 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:38:42,884 DEBUG lava-logs => PING(20) 2019-10-31 05:38:53,172 INFO scheduling health checks: 2019-10-31 05:38:53,249 INFO scheduling jobs: 2019-10-31 05:38:53,252 DEBUG - qemu 2019-10-31 05:38:53,331 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:38:53,331 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:38:53,365 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:38:54,652 INFO [INIT] Dropping privileges 2019-10-31 05:38:54,653 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:38:54,653 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:38:54,654 INFO [INIT] Marking all workers as offline 2019-10-31 05:38:54,685 INFO [INIT] LAVA master has started. 2019-10-31 05:38:54,685 INFO [INIT] Using protocol version 3 2019-10-31 05:38:54,849 DEBUG Dhanu => PING(20) 2019-10-31 05:38:54,849 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:39:02,904 DEBUG lava-logs => PING(20) 2019-10-31 05:39:14,695 INFO scheduling health checks: 2019-10-31 05:39:14,729 INFO scheduling jobs: 2019-10-31 05:39:14,732 DEBUG - qemu 2019-10-31 05:39:14,808 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:39:14,808 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:39:14,815 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:39:16,217 INFO [INIT] Dropping privileges 2019-10-31 05:39:16,217 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:39:16,218 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:39:16,219 INFO [INIT] Marking all workers as offline 2019-10-31 05:39:16,243 INFO [INIT] LAVA master has started. 2019-10-31 05:39:16,243 INFO [INIT] Using protocol version 3 2019-10-31 05:39:16,326 DEBUG Dhanu => PING(20) 2019-10-31 05:39:16,327 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:39:22,922 DEBUG lava-logs => PING(20) 2019-10-31 05:39:36,255 INFO scheduling health checks: 2019-10-31 05:39:36,286 INFO scheduling jobs: 2019-10-31 05:39:36,288 DEBUG - qemu 2019-10-31 05:39:36,362 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:39:36,362 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:39:36,367 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:39:37,648 INFO [INIT] Dropping privileges 2019-10-31 05:39:37,649 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:39:37,649 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:39:37,651 INFO [INIT] Marking all workers as offline 2019-10-31 05:39:37,698 INFO [INIT] LAVA master has started. 2019-10-31 05:39:37,699 INFO [INIT] Using protocol version 3 2019-10-31 05:39:42,937 DEBUG lava-logs => PING(20) 2019-10-31 05:39:56,360 DEBUG Dhanu => PING(20) 2019-10-31 05:39:56,361 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:39:57,699 INFO scheduling health checks: 2019-10-31 05:39:57,767 INFO scheduling jobs: 2019-10-31 05:39:57,771 DEBUG - qemu 2019-10-31 05:39:57,850 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:39:57,850 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:39:57,855 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:39:59,105 INFO [INIT] Dropping privileges 2019-10-31 05:39:59,105 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:39:59,105 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:39:59,107 INFO [INIT] Marking all workers as offline 2019-10-31 05:39:59,136 INFO [INIT] LAVA master has started. 2019-10-31 05:39:59,137 INFO [INIT] Using protocol version 3 2019-10-31 05:40:02,954 DEBUG lava-logs => PING(20) 2019-10-31 05:40:16,376 DEBUG Dhanu => PING(20) 2019-10-31 05:40:16,377 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:40:19,139 INFO scheduling health checks: 2019-10-31 05:40:19,180 INFO scheduling jobs: 2019-10-31 05:40:19,182 DEBUG - qemu 2019-10-31 05:40:19,264 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:40:19,265 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:40:19,270 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:40:20,804 INFO [INIT] Dropping privileges 2019-10-31 05:40:20,805 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:40:20,805 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:40:20,806 INFO [INIT] Marking all workers as offline 2019-10-31 05:40:20,837 INFO [INIT] LAVA master has started. 2019-10-31 05:40:20,837 INFO [INIT] Using protocol version 3 2019-10-31 05:40:22,966 DEBUG lava-logs => PING(20) 2019-10-31 05:40:36,392 DEBUG Dhanu => PING(20) 2019-10-31 05:40:36,393 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:40:40,839 INFO scheduling health checks: 2019-10-31 05:40:40,882 INFO scheduling jobs: 2019-10-31 05:40:40,884 DEBUG - qemu 2019-10-31 05:40:40,983 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:40:40,983 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:40:40,990 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:40:42,531 INFO [INIT] Dropping privileges 2019-10-31 05:40:42,531 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:40:42,531 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:40:42,533 INFO [INIT] Marking all workers as offline 2019-10-31 05:40:42,596 INFO [INIT] LAVA master has started. 2019-10-31 05:40:42,596 INFO [INIT] Using protocol version 3 2019-10-31 05:40:42,981 DEBUG lava-logs => PING(20) 2019-10-31 05:40:56,409 DEBUG Dhanu => PING(20) 2019-10-31 05:40:56,410 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:41:02,627 INFO scheduling health checks: 2019-10-31 05:41:02,674 INFO scheduling jobs: 2019-10-31 05:41:02,676 DEBUG - qemu 2019-10-31 05:41:02,767 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:41:02,767 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:41:02,812 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:41:04,279 INFO [INIT] Dropping privileges 2019-10-31 05:41:04,280 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:41:04,280 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:41:04,292 INFO [INIT] Marking all workers as offline 2019-10-31 05:41:04,320 INFO [INIT] LAVA master has started. 2019-10-31 05:41:04,320 INFO [INIT] Using protocol version 3 2019-10-31 05:41:04,390 DEBUG lava-logs => PING(20) 2019-10-31 05:41:16,428 DEBUG Dhanu => PING(20) 2019-10-31 05:41:16,428 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:41:24,334 INFO scheduling health checks: 2019-10-31 05:41:24,419 INFO scheduling jobs: 2019-10-31 05:41:24,423 DEBUG - qemu 2019-10-31 05:41:24,527 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:41:24,527 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:41:24,566 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:41:26,002 INFO [INIT] Dropping privileges 2019-10-31 05:41:26,003 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:41:26,003 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:41:26,005 INFO [INIT] Marking all workers as offline 2019-10-31 05:41:26,043 INFO [INIT] LAVA master has started. 2019-10-31 05:41:26,044 INFO [INIT] Using protocol version 3 2019-10-31 05:41:36,440 DEBUG Dhanu => PING(20) 2019-10-31 05:41:36,441 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:41:44,424 DEBUG lava-logs => PING(20) 2019-10-31 05:41:46,045 INFO scheduling health checks: 2019-10-31 05:41:46,077 INFO scheduling jobs: 2019-10-31 05:41:46,079 DEBUG - qemu 2019-10-31 05:41:46,152 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:41:46,152 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:41:46,161 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:41:47,322 INFO [INIT] Dropping privileges 2019-10-31 05:41:47,322 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:41:47,323 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:41:47,324 INFO [INIT] Marking all workers as offline 2019-10-31 05:41:47,368 INFO [INIT] LAVA master has started. 2019-10-31 05:41:47,369 INFO [INIT] Using protocol version 3 2019-10-31 05:41:56,456 DEBUG Dhanu => PING(20) 2019-10-31 05:41:56,457 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:42:04,442 DEBUG lava-logs => PING(20) 2019-10-31 05:42:07,371 INFO scheduling health checks: 2019-10-31 05:42:07,401 INFO scheduling jobs: 2019-10-31 05:42:07,403 DEBUG - qemu 2019-10-31 05:42:07,486 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:42:07,486 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:42:07,491 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:42:09,039 INFO [INIT] Dropping privileges 2019-10-31 05:42:09,040 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:42:09,040 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:42:09,042 INFO [INIT] Marking all workers as offline 2019-10-31 05:42:09,115 INFO [INIT] LAVA master has started. 2019-10-31 05:42:09,115 INFO [INIT] Using protocol version 3 2019-10-31 05:42:16,468 DEBUG Dhanu => PING(20) 2019-10-31 05:42:16,469 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:42:24,456 DEBUG lava-logs => PING(20) 2019-10-31 05:42:29,119 INFO scheduling health checks: 2019-10-31 05:42:29,159 INFO scheduling jobs: 2019-10-31 05:42:29,161 DEBUG - qemu 2019-10-31 05:42:29,238 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:42:29,238 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:42:29,243 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:42:30,477 INFO [INIT] Dropping privileges 2019-10-31 05:42:30,477 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:42:30,477 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:42:30,479 INFO [INIT] Marking all workers as offline 2019-10-31 05:42:30,516 INFO [INIT] LAVA master has started. 2019-10-31 05:42:30,517 INFO [INIT] Using protocol version 3 2019-10-31 05:42:36,480 DEBUG Dhanu => PING(20) 2019-10-31 05:42:36,481 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:42:44,468 DEBUG lava-logs => PING(20) 2019-10-31 05:42:50,520 INFO scheduling health checks: 2019-10-31 05:42:50,561 INFO scheduling jobs: 2019-10-31 05:42:50,563 DEBUG - qemu 2019-10-31 05:42:50,633 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:42:50,633 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:42:50,638 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:42:51,838 INFO [INIT] Dropping privileges 2019-10-31 05:42:51,838 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:42:51,838 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:42:51,840 INFO [INIT] Marking all workers as offline 2019-10-31 05:42:51,897 INFO [INIT] LAVA master has started. 2019-10-31 05:42:51,898 INFO [INIT] Using protocol version 3 2019-10-31 05:42:56,497 DEBUG Dhanu => PING(20) 2019-10-31 05:42:56,498 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:43:04,480 DEBUG lava-logs => PING(20) 2019-10-31 05:43:11,903 INFO scheduling health checks: 2019-10-31 05:43:11,941 INFO scheduling jobs: 2019-10-31 05:43:11,944 DEBUG - qemu 2019-10-31 05:43:12,028 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:43:12,028 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:43:12,033 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:43:13,495 INFO [INIT] Dropping privileges 2019-10-31 05:43:13,496 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:43:13,496 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:43:13,499 INFO [INIT] Marking all workers as offline 2019-10-31 05:43:13,533 INFO [INIT] LAVA master has started. 2019-10-31 05:43:13,533 INFO [INIT] Using protocol version 3 2019-10-31 05:43:16,514 DEBUG Dhanu => PING(20) 2019-10-31 05:43:16,514 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:43:24,492 DEBUG lava-logs => PING(20) 2019-10-31 05:43:33,557 INFO scheduling health checks: 2019-10-31 05:43:33,632 INFO scheduling jobs: 2019-10-31 05:43:33,636 DEBUG - qemu 2019-10-31 05:43:33,721 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:43:33,721 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:43:33,772 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:43:35,218 INFO [INIT] Dropping privileges 2019-10-31 05:43:35,218 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:43:35,218 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:43:35,220 INFO [INIT] Marking all workers as offline 2019-10-31 05:43:35,245 INFO [INIT] LAVA master has started. 2019-10-31 05:43:35,245 INFO [INIT] Using protocol version 3 2019-10-31 05:43:36,531 DEBUG Dhanu => PING(20) 2019-10-31 05:43:36,532 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:43:44,508 DEBUG lava-logs => PING(20) 2019-10-31 05:43:55,255 INFO scheduling health checks: 2019-10-31 05:43:55,289 INFO scheduling jobs: 2019-10-31 05:43:55,292 DEBUG - qemu 2019-10-31 05:43:55,370 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:43:55,370 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:43:55,376 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:43:56,795 INFO [INIT] Dropping privileges 2019-10-31 05:43:56,796 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:43:56,796 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:43:56,800 INFO [INIT] Marking all workers as offline 2019-10-31 05:43:56,835 INFO [INIT] LAVA master has started. 2019-10-31 05:43:56,836 INFO [INIT] Using protocol version 3 2019-10-31 05:43:56,897 DEBUG Dhanu => PING(20) 2019-10-31 05:43:56,897 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:44:04,529 DEBUG lava-logs => PING(20) 2019-10-31 05:44:16,848 INFO scheduling health checks: 2019-10-31 05:44:16,880 INFO scheduling jobs: 2019-10-31 05:44:16,883 DEBUG - qemu 2019-10-31 05:44:16,976 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:44:16,976 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:44:16,981 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:44:18,419 INFO [INIT] Dropping privileges 2019-10-31 05:44:18,420 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:44:18,420 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:44:18,421 INFO [INIT] Marking all workers as offline 2019-10-31 05:44:18,461 INFO [INIT] LAVA master has started. 2019-10-31 05:44:18,461 INFO [INIT] Using protocol version 3 2019-10-31 05:44:24,544 DEBUG lava-logs => PING(20) 2019-10-31 05:44:36,929 DEBUG Dhanu => PING(20) 2019-10-31 05:44:36,929 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:44:38,463 INFO scheduling health checks: 2019-10-31 05:44:38,482 INFO scheduling jobs: 2019-10-31 05:44:38,483 DEBUG - qemu 2019-10-31 05:44:38,552 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:44:38,553 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:44:38,558 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:44:39,903 INFO [INIT] Dropping privileges 2019-10-31 05:44:39,903 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:44:39,903 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:44:39,905 INFO [INIT] Marking all workers as offline 2019-10-31 05:44:39,950 INFO [INIT] LAVA master has started. 2019-10-31 05:44:39,951 INFO [INIT] Using protocol version 3 2019-10-31 05:44:44,560 DEBUG lava-logs => PING(20) 2019-10-31 05:44:56,944 DEBUG Dhanu => PING(20) 2019-10-31 05:44:56,945 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:44:59,952 INFO scheduling health checks: 2019-10-31 05:44:59,995 INFO scheduling jobs: 2019-10-31 05:44:59,997 DEBUG - qemu 2019-10-31 05:45:00,093 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:45:00,093 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:45:00,100 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:45:01,482 INFO [INIT] Dropping privileges 2019-10-31 05:45:01,482 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:45:01,482 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:45:01,484 INFO [INIT] Marking all workers as offline 2019-10-31 05:45:01,521 INFO [INIT] LAVA master has started. 2019-10-31 05:45:01,521 INFO [INIT] Using protocol version 3 2019-10-31 05:45:04,575 DEBUG lava-logs => PING(20) 2019-10-31 05:45:16,960 DEBUG Dhanu => PING(20) 2019-10-31 05:45:16,961 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:45:21,523 INFO scheduling health checks: 2019-10-31 05:45:21,553 INFO scheduling jobs: 2019-10-31 05:45:21,555 DEBUG - qemu 2019-10-31 05:45:21,629 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:45:21,629 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:45:21,633 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:45:22,822 INFO [INIT] Dropping privileges 2019-10-31 05:45:22,822 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:45:22,822 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:45:22,824 INFO [INIT] Marking all workers as offline 2019-10-31 05:45:22,878 INFO [INIT] LAVA master has started. 2019-10-31 05:45:22,879 INFO [INIT] Using protocol version 3 2019-10-31 05:45:24,586 DEBUG lava-logs => PING(20) 2019-10-31 05:45:36,976 DEBUG Dhanu => PING(20) 2019-10-31 05:45:36,977 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:45:42,883 INFO scheduling health checks: 2019-10-31 05:45:42,922 INFO scheduling jobs: 2019-10-31 05:45:42,924 DEBUG - qemu 2019-10-31 05:45:42,998 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:45:42,998 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:45:43,003 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:45:44,346 INFO [INIT] Dropping privileges 2019-10-31 05:45:44,346 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:45:44,346 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:45:44,348 INFO [INIT] Marking all workers as offline 2019-10-31 05:45:44,380 INFO [INIT] LAVA master has started. 2019-10-31 05:45:44,381 INFO [INIT] Using protocol version 3 2019-10-31 05:45:44,596 DEBUG lava-logs => PING(20) 2019-10-31 05:45:56,989 DEBUG Dhanu => PING(20) 2019-10-31 05:45:56,990 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:46:04,387 INFO scheduling health checks: 2019-10-31 05:46:04,446 INFO scheduling jobs: 2019-10-31 05:46:04,449 DEBUG - qemu 2019-10-31 05:46:04,523 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:46:04,524 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:46:04,528 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:46:05,812 INFO [INIT] Dropping privileges 2019-10-31 05:46:05,812 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:46:05,812 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:46:05,814 INFO [INIT] Marking all workers as offline 2019-10-31 05:46:05,860 INFO [INIT] LAVA master has started. 2019-10-31 05:46:05,861 INFO [INIT] Using protocol version 3 2019-10-31 05:46:06,045 DEBUG lava-logs => PING(20) 2019-10-31 05:46:17,005 DEBUG Dhanu => PING(20) 2019-10-31 05:46:17,006 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:46:25,867 INFO scheduling health checks: 2019-10-31 05:46:25,904 INFO scheduling jobs: 2019-10-31 05:46:25,905 DEBUG - qemu 2019-10-31 05:46:25,980 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:46:25,980 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:46:25,985 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:46:27,325 INFO [INIT] Dropping privileges 2019-10-31 05:46:27,326 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:46:27,326 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:46:27,327 INFO [INIT] Marking all workers as offline 2019-10-31 05:46:27,362 INFO [INIT] LAVA master has started. 2019-10-31 05:46:27,363 INFO [INIT] Using protocol version 3 2019-10-31 05:46:27,507 DEBUG lava-logs => PING(20) 2019-10-31 05:46:37,024 DEBUG Dhanu => PING(20) 2019-10-31 05:46:37,025 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:46:47,372 INFO scheduling health checks: 2019-10-31 05:46:47,395 INFO scheduling jobs: 2019-10-31 05:46:47,397 DEBUG - qemu 2019-10-31 05:46:47,471 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:46:47,471 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:46:47,476 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:46:48,572 INFO [INIT] Dropping privileges 2019-10-31 05:46:48,572 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:46:48,572 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:46:48,574 INFO [INIT] Marking all workers as offline 2019-10-31 05:46:48,608 INFO [INIT] LAVA master has started. 2019-10-31 05:46:48,608 INFO [INIT] Using protocol version 3 2019-10-31 05:46:48,725 DEBUG lava-logs => PING(20) 2019-10-31 05:46:57,040 DEBUG Dhanu => PING(20) 2019-10-31 05:46:57,041 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:47:08,619 INFO scheduling health checks: 2019-10-31 05:47:08,652 INFO scheduling jobs: 2019-10-31 05:47:08,655 DEBUG - qemu 2019-10-31 05:47:08,727 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:47:08,728 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:47:08,736 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:47:10,073 INFO [INIT] Dropping privileges 2019-10-31 05:47:10,073 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:47:10,073 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:47:10,075 INFO [INIT] Marking all workers as offline 2019-10-31 05:47:10,122 INFO [INIT] LAVA master has started. 2019-10-31 05:47:10,122 INFO [INIT] Using protocol version 3 2019-10-31 05:47:10,190 DEBUG lava-logs => PING(20) 2019-10-31 05:47:17,052 DEBUG Dhanu => PING(20) 2019-10-31 05:47:17,053 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:47:30,135 INFO scheduling health checks: 2019-10-31 05:47:30,177 INFO scheduling jobs: 2019-10-31 05:47:30,180 DEBUG - qemu 2019-10-31 05:47:30,260 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:47:30,260 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:47:30,265 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:47:31,578 INFO [INIT] Dropping privileges 2019-10-31 05:47:31,578 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:47:31,578 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:47:31,580 INFO [INIT] Marking all workers as offline 2019-10-31 05:47:31,613 INFO [INIT] LAVA master has started. 2019-10-31 05:47:31,613 INFO [INIT] Using protocol version 3 2019-10-31 05:47:37,064 DEBUG Dhanu => PING(20) 2019-10-31 05:47:37,065 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:47:50,226 DEBUG lava-logs => PING(20) 2019-10-31 05:47:51,615 INFO scheduling health checks: 2019-10-31 05:47:51,634 INFO scheduling jobs: 2019-10-31 05:47:51,636 DEBUG - qemu 2019-10-31 05:47:51,704 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:47:51,705 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:47:51,709 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:47:52,848 INFO [INIT] Dropping privileges 2019-10-31 05:47:52,848 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:47:52,849 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:47:52,850 INFO [INIT] Marking all workers as offline 2019-10-31 05:47:52,892 INFO [INIT] LAVA master has started. 2019-10-31 05:47:52,892 INFO [INIT] Using protocol version 3 2019-10-31 05:47:57,080 DEBUG Dhanu => PING(20) 2019-10-31 05:47:57,080 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:48:10,244 DEBUG lava-logs => PING(20) 2019-10-31 05:48:12,895 INFO scheduling health checks: 2019-10-31 05:48:12,936 INFO scheduling jobs: 2019-10-31 05:48:12,939 DEBUG - qemu 2019-10-31 05:48:13,019 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:48:13,020 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:48:13,024 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:48:14,364 INFO [INIT] Dropping privileges 2019-10-31 05:48:14,365 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:48:14,365 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:48:14,367 INFO [INIT] Marking all workers as offline 2019-10-31 05:48:14,439 INFO [INIT] LAVA master has started. 2019-10-31 05:48:14,440 INFO [INIT] Using protocol version 3 2019-10-31 05:48:17,095 DEBUG Dhanu => PING(20) 2019-10-31 05:48:17,095 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:48:30,256 DEBUG lava-logs => PING(20) 2019-10-31 05:48:34,443 INFO scheduling health checks: 2019-10-31 05:48:34,483 INFO scheduling jobs: 2019-10-31 05:48:34,486 DEBUG - qemu 2019-10-31 05:48:34,562 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:48:34,562 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:48:34,567 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:48:35,828 INFO [INIT] Dropping privileges 2019-10-31 05:48:35,828 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:48:35,828 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:48:35,829 INFO [INIT] Marking all workers as offline 2019-10-31 05:48:35,863 INFO [INIT] LAVA master has started. 2019-10-31 05:48:35,863 INFO [INIT] Using protocol version 3 2019-10-31 05:48:37,110 DEBUG Dhanu => PING(20) 2019-10-31 05:48:37,111 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:48:50,268 DEBUG lava-logs => PING(20) 2019-10-31 05:48:55,867 INFO scheduling health checks: 2019-10-31 05:48:55,907 INFO scheduling jobs: 2019-10-31 05:48:55,910 DEBUG - qemu 2019-10-31 05:48:55,986 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:48:55,987 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:48:55,991 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:48:57,315 INFO [INIT] Dropping privileges 2019-10-31 05:48:57,316 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:48:57,316 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:48:57,317 INFO [INIT] Marking all workers as offline 2019-10-31 05:48:57,354 INFO [INIT] LAVA master has started. 2019-10-31 05:48:57,354 INFO [INIT] Using protocol version 3 2019-10-31 05:48:57,432 DEBUG Dhanu => PING(20) 2019-10-31 05:48:57,432 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:49:10,280 DEBUG lava-logs => PING(20) 2019-10-31 05:49:17,359 INFO scheduling health checks: 2019-10-31 05:49:17,398 INFO scheduling jobs: 2019-10-31 05:49:17,401 DEBUG - qemu 2019-10-31 05:49:17,476 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:49:17,476 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:49:17,484 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:49:18,821 INFO [INIT] Dropping privileges 2019-10-31 05:49:18,821 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:49:18,821 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:49:18,822 INFO [INIT] Marking all workers as offline 2019-10-31 05:49:18,856 INFO [INIT] LAVA master has started. 2019-10-31 05:49:18,856 INFO [INIT] Using protocol version 3 2019-10-31 05:49:30,292 DEBUG lava-logs => PING(20) 2019-10-31 05:49:37,464 DEBUG Dhanu => PING(20) 2019-10-31 05:49:37,465 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:49:38,858 INFO scheduling health checks: 2019-10-31 05:49:38,911 INFO scheduling jobs: 2019-10-31 05:49:38,913 DEBUG - qemu 2019-10-31 05:49:38,989 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:49:38,989 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:49:38,994 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:49:40,308 INFO [INIT] Dropping privileges 2019-10-31 05:49:40,309 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:49:40,309 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:49:40,310 INFO [INIT] Marking all workers as offline 2019-10-31 05:49:40,358 INFO [INIT] LAVA master has started. 2019-10-31 05:49:40,359 INFO [INIT] Using protocol version 3 2019-10-31 05:49:50,312 DEBUG lava-logs => PING(20) 2019-10-31 05:49:57,480 DEBUG Dhanu => PING(20) 2019-10-31 05:49:57,481 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:50:00,361 INFO scheduling health checks: 2019-10-31 05:50:00,391 INFO scheduling jobs: 2019-10-31 05:50:00,393 DEBUG - qemu 2019-10-31 05:50:00,469 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:50:00,469 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:50:00,474 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:50:01,840 INFO [INIT] Dropping privileges 2019-10-31 05:50:01,840 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:50:01,840 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:50:01,842 INFO [INIT] Marking all workers as offline 2019-10-31 05:50:01,894 INFO [INIT] LAVA master has started. 2019-10-31 05:50:01,895 INFO [INIT] Using protocol version 3 2019-10-31 05:50:10,332 DEBUG lava-logs => PING(20) 2019-10-31 05:50:17,496 DEBUG Dhanu => PING(20) 2019-10-31 05:50:17,497 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:50:21,899 INFO scheduling health checks: 2019-10-31 05:50:21,960 INFO scheduling jobs: 2019-10-31 05:50:21,962 DEBUG - qemu 2019-10-31 05:50:22,034 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:50:22,034 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:50:22,039 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:50:23,312 INFO [INIT] Dropping privileges 2019-10-31 05:50:23,313 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:50:23,313 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:50:23,314 INFO [INIT] Marking all workers as offline 2019-10-31 05:50:23,351 INFO [INIT] LAVA master has started. 2019-10-31 05:50:23,351 INFO [INIT] Using protocol version 3 2019-10-31 05:50:30,350 DEBUG lava-logs => PING(20) 2019-10-31 05:50:37,512 DEBUG Dhanu => PING(20) 2019-10-31 05:50:37,513 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:50:43,355 INFO scheduling health checks: 2019-10-31 05:50:43,406 INFO scheduling jobs: 2019-10-31 05:50:43,410 DEBUG - qemu 2019-10-31 05:50:43,487 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:50:43,487 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:50:43,492 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:50:44,823 INFO [INIT] Dropping privileges 2019-10-31 05:50:44,823 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:50:44,823 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:50:44,825 INFO [INIT] Marking all workers as offline 2019-10-31 05:50:44,864 INFO [INIT] LAVA master has started. 2019-10-31 05:50:44,865 INFO [INIT] Using protocol version 3 2019-10-31 05:50:50,364 DEBUG lava-logs => PING(20) 2019-10-31 05:50:57,528 DEBUG Dhanu => PING(20) 2019-10-31 05:50:57,529 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:51:04,871 INFO scheduling health checks: 2019-10-31 05:51:04,920 INFO scheduling jobs: 2019-10-31 05:51:04,923 DEBUG - qemu 2019-10-31 05:51:05,006 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:51:05,007 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:51:05,011 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:51:06,331 INFO [INIT] Dropping privileges 2019-10-31 05:51:06,331 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:51:06,331 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:51:06,333 INFO [INIT] Marking all workers as offline 2019-10-31 05:51:06,402 INFO [INIT] LAVA master has started. 2019-10-31 05:51:06,403 INFO [INIT] Using protocol version 3 2019-10-31 05:51:10,380 DEBUG lava-logs => PING(20) 2019-10-31 05:51:17,540 DEBUG Dhanu => PING(20) 2019-10-31 05:51:17,541 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:51:26,411 INFO scheduling health checks: 2019-10-31 05:51:26,466 INFO scheduling jobs: 2019-10-31 05:51:26,469 DEBUG - qemu 2019-10-31 05:51:26,546 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:51:26,546 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:51:26,550 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:51:27,845 INFO [INIT] Dropping privileges 2019-10-31 05:51:27,845 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:51:27,845 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:51:27,847 INFO [INIT] Marking all workers as offline 2019-10-31 05:51:27,879 INFO [INIT] LAVA master has started. 2019-10-31 05:51:27,879 INFO [INIT] Using protocol version 3 2019-10-31 05:51:30,392 DEBUG lava-logs => PING(20) 2019-10-31 05:51:37,556 DEBUG Dhanu => PING(20) 2019-10-31 05:51:37,557 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:51:47,888 INFO scheduling health checks: 2019-10-31 05:51:47,923 INFO scheduling jobs: 2019-10-31 05:51:47,925 DEBUG - qemu 2019-10-31 05:51:48,001 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:51:48,001 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:51:48,006 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:51:49,336 INFO [INIT] Dropping privileges 2019-10-31 05:51:49,336 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:51:49,336 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:51:49,338 INFO [INIT] Marking all workers as offline 2019-10-31 05:51:49,393 INFO [INIT] LAVA master has started. 2019-10-31 05:51:49,394 INFO [INIT] Using protocol version 3 2019-10-31 05:51:50,406 DEBUG lava-logs => PING(20) 2019-10-31 05:51:57,572 DEBUG Dhanu => PING(20) 2019-10-31 05:51:57,573 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:52:09,403 INFO scheduling health checks: 2019-10-31 05:52:09,459 INFO scheduling jobs: 2019-10-31 05:52:09,462 DEBUG - qemu 2019-10-31 05:52:09,539 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:52:09,539 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:52:09,543 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:52:10,826 INFO [INIT] Dropping privileges 2019-10-31 05:52:10,827 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:52:10,827 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:52:10,828 INFO [INIT] Marking all workers as offline 2019-10-31 05:52:10,872 INFO [INIT] LAVA master has started. 2019-10-31 05:52:10,873 INFO [INIT] Using protocol version 3 2019-10-31 05:52:10,958 DEBUG lava-logs => PING(20) 2019-10-31 05:52:17,584 DEBUG Dhanu => PING(20) 2019-10-31 05:52:17,585 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:52:30,883 INFO scheduling health checks: 2019-10-31 05:52:30,939 INFO scheduling jobs: 2019-10-31 05:52:30,942 DEBUG - qemu 2019-10-31 05:52:31,015 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:52:31,015 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:52:31,020 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:52:32,369 INFO [INIT] Dropping privileges 2019-10-31 05:52:32,369 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:52:32,369 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:52:32,371 INFO [INIT] Marking all workers as offline 2019-10-31 05:52:32,419 INFO [INIT] LAVA master has started. 2019-10-31 05:52:32,419 INFO [INIT] Using protocol version 3 2019-10-31 05:52:37,596 DEBUG Dhanu => PING(20) 2019-10-31 05:52:37,597 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:52:50,992 DEBUG lava-logs => PING(20) 2019-10-31 05:52:52,421 INFO scheduling health checks: 2019-10-31 05:52:52,452 INFO scheduling jobs: 2019-10-31 05:52:52,454 DEBUG - qemu 2019-10-31 05:52:52,528 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:52:52,528 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:52:52,533 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:52:53,844 INFO [INIT] Dropping privileges 2019-10-31 05:52:53,844 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:52:53,844 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:52:53,846 INFO [INIT] Marking all workers as offline 2019-10-31 05:52:53,876 INFO [INIT] LAVA master has started. 2019-10-31 05:52:53,877 INFO [INIT] Using protocol version 3 2019-10-31 05:52:57,613 DEBUG Dhanu => PING(20) 2019-10-31 05:52:57,613 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:53:11,008 DEBUG lava-logs => PING(20) 2019-10-31 05:53:13,879 INFO scheduling health checks: 2019-10-31 05:53:13,909 INFO scheduling jobs: 2019-10-31 05:53:13,911 DEBUG - qemu 2019-10-31 05:53:13,984 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:53:13,984 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:53:13,989 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:53:15,357 INFO [INIT] Dropping privileges 2019-10-31 05:53:15,357 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:53:15,357 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:53:15,359 INFO [INIT] Marking all workers as offline 2019-10-31 05:53:15,401 INFO [INIT] LAVA master has started. 2019-10-31 05:53:15,402 INFO [INIT] Using protocol version 3 2019-10-31 05:53:17,627 DEBUG Dhanu => PING(20) 2019-10-31 05:53:17,627 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:53:31,024 DEBUG lava-logs => PING(20) 2019-10-31 05:53:35,403 INFO scheduling health checks: 2019-10-31 05:53:35,434 INFO scheduling jobs: 2019-10-31 05:53:35,436 DEBUG - qemu 2019-10-31 05:53:35,510 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:53:35,510 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:53:35,518 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:53:36,848 INFO [INIT] Dropping privileges 2019-10-31 05:53:36,848 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:53:36,848 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:53:36,850 INFO [INIT] Marking all workers as offline 2019-10-31 05:53:36,892 INFO [INIT] LAVA master has started. 2019-10-31 05:53:36,893 INFO [INIT] Using protocol version 3 2019-10-31 05:53:37,641 DEBUG Dhanu => PING(20) 2019-10-31 05:53:37,642 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:53:51,036 DEBUG lava-logs => PING(20) 2019-10-31 05:53:56,895 INFO scheduling health checks: 2019-10-31 05:53:56,947 INFO scheduling jobs: 2019-10-31 05:53:56,950 DEBUG - qemu 2019-10-31 05:53:57,024 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:53:57,024 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:53:57,029 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:53:58,334 INFO [INIT] Dropping privileges 2019-10-31 05:53:58,334 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:53:58,334 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:53:58,336 INFO [INIT] Marking all workers as offline 2019-10-31 05:53:58,372 INFO [INIT] LAVA master has started. 2019-10-31 05:53:58,372 INFO [INIT] Using protocol version 3 2019-10-31 05:53:58,489 DEBUG Dhanu => PING(20) 2019-10-31 05:53:58,489 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:54:11,048 DEBUG lava-logs => PING(20) 2019-10-31 05:54:18,379 INFO scheduling health checks: 2019-10-31 05:54:18,405 INFO scheduling jobs: 2019-10-31 05:54:18,407 DEBUG - qemu 2019-10-31 05:54:18,483 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:54:18,483 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:54:18,488 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:54:19,838 INFO [INIT] Dropping privileges 2019-10-31 05:54:19,838 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:54:19,838 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:54:19,840 INFO [INIT] Marking all workers as offline 2019-10-31 05:54:19,873 INFO [INIT] LAVA master has started. 2019-10-31 05:54:19,874 INFO [INIT] Using protocol version 3 2019-10-31 05:54:20,010 DEBUG Dhanu => PING(20) 2019-10-31 05:54:20,010 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:54:31,060 DEBUG lava-logs => PING(20) 2019-10-31 05:54:39,879 INFO scheduling health checks: 2019-10-31 05:54:39,929 INFO scheduling jobs: 2019-10-31 05:54:39,931 DEBUG - qemu 2019-10-31 05:54:40,004 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:54:40,005 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:54:40,014 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:54:41,333 INFO [INIT] Dropping privileges 2019-10-31 05:54:41,333 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:54:41,333 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:54:41,335 INFO [INIT] Marking all workers as offline 2019-10-31 05:54:41,387 INFO [INIT] LAVA master has started. 2019-10-31 05:54:41,388 INFO [INIT] Using protocol version 3 2019-10-31 05:54:41,516 DEBUG Dhanu => PING(20) 2019-10-31 05:54:41,516 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:54:51,100 DEBUG lava-logs => PING(20) 2019-10-31 05:55:01,395 INFO scheduling health checks: 2019-10-31 05:55:01,431 INFO scheduling jobs: 2019-10-31 05:55:01,433 DEBUG - qemu 2019-10-31 05:55:01,503 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:55:01,503 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:55:01,511 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:55:02,835 INFO [INIT] Dropping privileges 2019-10-31 05:55:02,835 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:55:02,835 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:55:02,837 INFO [INIT] Marking all workers as offline 2019-10-31 05:55:02,878 INFO [INIT] LAVA master has started. 2019-10-31 05:55:02,878 INFO [INIT] Using protocol version 3 2019-10-31 05:55:02,991 DEBUG Dhanu => PING(20) 2019-10-31 05:55:02,992 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:55:11,120 DEBUG lava-logs => PING(20) 2019-10-31 05:55:22,887 INFO scheduling health checks: 2019-10-31 05:55:22,911 INFO scheduling jobs: 2019-10-31 05:55:22,913 DEBUG - qemu 2019-10-31 05:55:22,987 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:55:22,988 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:55:22,996 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:55:24,327 INFO [INIT] Dropping privileges 2019-10-31 05:55:24,327 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:55:24,327 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:55:24,329 INFO [INIT] Marking all workers as offline 2019-10-31 05:55:24,369 INFO [INIT] LAVA master has started. 2019-10-31 05:55:24,369 INFO [INIT] Using protocol version 3 2019-10-31 05:55:24,387 DEBUG Dhanu => PING(20) 2019-10-31 05:55:24,387 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:55:31,136 DEBUG lava-logs => PING(20) 2019-10-31 05:55:44,379 INFO scheduling health checks: 2019-10-31 05:55:44,413 INFO scheduling jobs: 2019-10-31 05:55:44,414 DEBUG - qemu 2019-10-31 05:55:44,485 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:55:44,485 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:55:44,490 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:55:45,826 INFO [INIT] Dropping privileges 2019-10-31 05:55:45,826 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:55:45,826 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:55:45,828 INFO [INIT] Marking all workers as offline 2019-10-31 05:55:45,871 INFO [INIT] LAVA master has started. 2019-10-31 05:55:45,872 INFO [INIT] Using protocol version 3 2019-10-31 05:55:51,152 DEBUG lava-logs => PING(20) 2019-10-31 05:56:04,421 DEBUG Dhanu => PING(20) 2019-10-31 05:56:04,421 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:56:05,874 INFO scheduling health checks: 2019-10-31 05:56:05,904 INFO scheduling jobs: 2019-10-31 05:56:05,906 DEBUG - qemu 2019-10-31 05:56:05,987 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:56:05,987 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:56:05,992 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:56:07,345 INFO [INIT] Dropping privileges 2019-10-31 05:56:07,345 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:56:07,345 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:56:07,347 INFO [INIT] Marking all workers as offline 2019-10-31 05:56:07,384 INFO [INIT] LAVA master has started. 2019-10-31 05:56:07,385 INFO [INIT] Using protocol version 3 2019-10-31 05:56:11,168 DEBUG lava-logs => PING(20) 2019-10-31 05:56:24,436 DEBUG Dhanu => PING(20) 2019-10-31 05:56:24,437 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:56:27,387 INFO scheduling health checks: 2019-10-31 05:56:27,417 INFO scheduling jobs: 2019-10-31 05:56:27,419 DEBUG - qemu 2019-10-31 05:56:27,498 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:56:27,498 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:56:27,503 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:56:28,851 INFO [INIT] Dropping privileges 2019-10-31 05:56:28,851 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:56:28,851 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:56:28,853 INFO [INIT] Marking all workers as offline 2019-10-31 05:56:28,897 INFO [INIT] LAVA master has started. 2019-10-31 05:56:28,897 INFO [INIT] Using protocol version 3 2019-10-31 05:56:31,182 DEBUG lava-logs => PING(20) 2019-10-31 05:56:44,452 DEBUG Dhanu => PING(20) 2019-10-31 05:56:44,453 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:56:48,899 INFO scheduling health checks: 2019-10-31 05:56:48,919 INFO scheduling jobs: 2019-10-31 05:56:48,920 DEBUG - qemu 2019-10-31 05:56:48,989 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:56:48,989 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:56:48,994 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:56:50,345 INFO [INIT] Dropping privileges 2019-10-31 05:56:50,345 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:56:50,346 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:56:50,347 INFO [INIT] Marking all workers as offline 2019-10-31 05:56:50,388 INFO [INIT] LAVA master has started. 2019-10-31 05:56:50,388 INFO [INIT] Using protocol version 3 2019-10-31 05:56:51,193 DEBUG lava-logs => PING(20) 2019-10-31 05:57:04,469 DEBUG Dhanu => PING(20) 2019-10-31 05:57:04,469 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:57:10,391 INFO scheduling health checks: 2019-10-31 05:57:10,432 INFO scheduling jobs: 2019-10-31 05:57:10,435 DEBUG - qemu 2019-10-31 05:57:10,518 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:57:10,518 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:57:10,523 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:57:11,845 INFO [INIT] Dropping privileges 2019-10-31 05:57:11,845 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:57:11,845 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:57:11,847 INFO [INIT] Marking all workers as offline 2019-10-31 05:57:11,901 INFO [INIT] LAVA master has started. 2019-10-31 05:57:11,902 INFO [INIT] Using protocol version 3 2019-10-31 05:57:11,966 DEBUG lava-logs => PING(20) 2019-10-31 05:57:24,488 DEBUG Dhanu => PING(20) 2019-10-31 05:57:24,489 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:57:31,909 INFO scheduling health checks: 2019-10-31 05:57:31,947 INFO scheduling jobs: 2019-10-31 05:57:31,949 DEBUG - qemu 2019-10-31 05:57:32,040 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:57:32,040 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:57:32,045 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:57:33,390 INFO [INIT] Dropping privileges 2019-10-31 05:57:33,390 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:57:33,390 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:57:33,392 INFO [INIT] Marking all workers as offline 2019-10-31 05:57:33,425 INFO [INIT] LAVA master has started. 2019-10-31 05:57:33,427 INFO [INIT] Using protocol version 3 2019-10-31 05:57:44,500 DEBUG Dhanu => PING(20) 2019-10-31 05:57:44,500 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:57:52,003 DEBUG lava-logs => PING(20) 2019-10-31 05:57:53,428 INFO scheduling health checks: 2019-10-31 05:57:53,459 INFO scheduling jobs: 2019-10-31 05:57:53,461 DEBUG - qemu 2019-10-31 05:57:53,553 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:57:53,553 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:57:53,559 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:57:54,982 INFO [INIT] Dropping privileges 2019-10-31 05:57:54,983 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:57:54,983 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:57:54,984 INFO [INIT] Marking all workers as offline 2019-10-31 05:57:55,016 INFO [INIT] LAVA master has started. 2019-10-31 05:57:55,016 INFO [INIT] Using protocol version 3 2019-10-31 05:58:04,519 DEBUG Dhanu => PING(20) 2019-10-31 05:58:04,519 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:58:12,023 DEBUG lava-logs => PING(20) 2019-10-31 05:58:15,020 INFO scheduling health checks: 2019-10-31 05:58:15,061 INFO scheduling jobs: 2019-10-31 05:58:15,063 DEBUG - qemu 2019-10-31 05:58:15,138 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:58:15,139 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:58:15,143 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:58:16,289 INFO [INIT] Dropping privileges 2019-10-31 05:58:16,290 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:58:16,290 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:58:16,291 INFO [INIT] Marking all workers as offline 2019-10-31 05:58:16,329 INFO [INIT] LAVA master has started. 2019-10-31 05:58:16,330 INFO [INIT] Using protocol version 3 2019-10-31 05:58:24,541 DEBUG Dhanu => PING(20) 2019-10-31 05:58:24,542 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:58:32,032 DEBUG lava-logs => PING(20) 2019-10-31 05:58:36,334 INFO scheduling health checks: 2019-10-31 05:58:36,374 INFO scheduling jobs: 2019-10-31 05:58:36,377 DEBUG - qemu 2019-10-31 05:58:36,450 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:58:36,451 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:58:36,455 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:58:37,558 INFO [INIT] Dropping privileges 2019-10-31 05:58:37,558 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:58:37,558 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:58:37,560 INFO [INIT] Marking all workers as offline 2019-10-31 05:58:37,621 INFO [INIT] LAVA master has started. 2019-10-31 05:58:37,621 INFO [INIT] Using protocol version 3 2019-10-31 05:58:44,560 DEBUG Dhanu => PING(20) 2019-10-31 05:58:44,561 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:58:52,048 DEBUG lava-logs => PING(20) 2019-10-31 05:58:57,627 INFO scheduling health checks: 2019-10-31 05:58:57,665 INFO scheduling jobs: 2019-10-31 05:58:57,667 DEBUG - qemu 2019-10-31 05:58:57,736 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:58:57,736 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:58:57,741 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:58:59,034 INFO [INIT] Dropping privileges 2019-10-31 05:58:59,035 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:58:59,035 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:58:59,036 INFO [INIT] Marking all workers as offline 2019-10-31 05:58:59,079 INFO [INIT] LAVA master has started. 2019-10-31 05:58:59,079 INFO [INIT] Using protocol version 3 2019-10-31 05:59:04,579 DEBUG Dhanu => PING(20) 2019-10-31 05:59:04,579 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:59:12,060 DEBUG lava-logs => PING(20) 2019-10-31 05:59:19,087 INFO scheduling health checks: 2019-10-31 05:59:19,134 INFO scheduling jobs: 2019-10-31 05:59:19,137 DEBUG - qemu 2019-10-31 05:59:19,211 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:59:19,211 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:59:19,216 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:59:20,296 INFO [INIT] Dropping privileges 2019-10-31 05:59:20,296 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:59:20,296 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:59:20,298 INFO [INIT] Marking all workers as offline 2019-10-31 05:59:20,336 INFO [INIT] LAVA master has started. 2019-10-31 05:59:20,336 INFO [INIT] Using protocol version 3 2019-10-31 05:59:24,600 DEBUG Dhanu => PING(20) 2019-10-31 05:59:24,601 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:59:32,070 DEBUG lava-logs => PING(20) 2019-10-31 05:59:40,345 INFO scheduling health checks: 2019-10-31 05:59:40,380 INFO scheduling jobs: 2019-10-31 05:59:40,384 DEBUG - qemu 2019-10-31 05:59:40,456 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 05:59:40,456 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 05:59:40,461 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 05:59:41,524 INFO [INIT] Dropping privileges 2019-10-31 05:59:41,524 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 05:59:41,525 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 05:59:41,526 INFO [INIT] Marking all workers as offline 2019-10-31 05:59:41,560 INFO [INIT] LAVA master has started. 2019-10-31 05:59:41,560 INFO [INIT] Using protocol version 3 2019-10-31 05:59:44,623 DEBUG Dhanu => PING(20) 2019-10-31 05:59:44,624 WARNING Unknown dispatcher (server crashed) 2019-10-31 05:59:52,084 DEBUG lava-logs => PING(20) 2019-10-31 06:00:01,569 INFO scheduling health checks: 2019-10-31 06:00:01,604 INFO scheduling jobs: 2019-10-31 06:00:01,607 DEBUG - qemu 2019-10-31 06:00:01,677 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:00:01,677 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:00:01,682 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:00:02,783 INFO [INIT] Dropping privileges 2019-10-31 06:00:02,783 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:00:02,783 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:00:02,785 INFO [INIT] Marking all workers as offline 2019-10-31 06:00:02,841 INFO [INIT] LAVA master has started. 2019-10-31 06:00:02,841 INFO [INIT] Using protocol version 3 2019-10-31 06:00:04,644 DEBUG Dhanu => PING(20) 2019-10-31 06:00:04,645 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:00:12,102 DEBUG lava-logs => PING(20) 2019-10-31 06:00:22,852 INFO scheduling health checks: 2019-10-31 06:00:22,884 INFO scheduling jobs: 2019-10-31 06:00:22,887 DEBUG - qemu 2019-10-31 06:00:22,961 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:00:22,962 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:00:22,972 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:00:24,030 INFO [INIT] Dropping privileges 2019-10-31 06:00:24,030 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:00:24,031 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:00:24,032 INFO [INIT] Marking all workers as offline 2019-10-31 06:00:24,098 INFO [INIT] LAVA master has started. 2019-10-31 06:00:24,098 INFO [INIT] Using protocol version 3 2019-10-31 06:00:24,664 DEBUG Dhanu => PING(20) 2019-10-31 06:00:24,665 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:00:32,121 DEBUG lava-logs => PING(20) 2019-10-31 06:00:44,110 INFO scheduling health checks: 2019-10-31 06:00:44,142 INFO scheduling jobs: 2019-10-31 06:00:44,143 DEBUG - qemu 2019-10-31 06:00:44,209 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:00:44,210 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:00:44,214 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:00:45,275 INFO [INIT] Dropping privileges 2019-10-31 06:00:45,276 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:00:45,276 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:00:45,277 INFO [INIT] Marking all workers as offline 2019-10-31 06:00:45,345 INFO [INIT] LAVA master has started. 2019-10-31 06:00:45,345 INFO [INIT] Using protocol version 3 2019-10-31 06:00:45,364 DEBUG Dhanu => PING(20) 2019-10-31 06:00:45,364 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:00:52,140 DEBUG lava-logs => PING(20) 2019-10-31 06:01:05,359 INFO scheduling health checks: 2019-10-31 06:01:05,399 INFO scheduling jobs: 2019-10-31 06:01:05,401 DEBUG - qemu 2019-10-31 06:01:05,472 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:01:05,472 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:01:05,477 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:01:06,538 INFO [INIT] Dropping privileges 2019-10-31 06:01:06,538 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:01:06,539 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:01:06,541 INFO [INIT] Marking all workers as offline 2019-10-31 06:01:06,579 INFO [INIT] LAVA master has started. 2019-10-31 06:01:06,580 INFO [INIT] Using protocol version 3 2019-10-31 06:01:12,158 DEBUG lava-logs => PING(20) 2019-10-31 06:01:25,406 DEBUG Dhanu => PING(20) 2019-10-31 06:01:25,407 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:01:26,581 INFO scheduling health checks: 2019-10-31 06:01:26,601 INFO scheduling jobs: 2019-10-31 06:01:26,603 DEBUG - qemu 2019-10-31 06:01:26,673 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:01:26,674 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:01:26,678 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:01:27,780 INFO [INIT] Dropping privileges 2019-10-31 06:01:27,780 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:01:27,780 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:01:27,782 INFO [INIT] Marking all workers as offline 2019-10-31 06:01:27,815 INFO [INIT] LAVA master has started. 2019-10-31 06:01:27,815 INFO [INIT] Using protocol version 3 2019-10-31 06:01:32,172 DEBUG lava-logs => PING(20) 2019-10-31 06:01:45,427 DEBUG Dhanu => PING(20) 2019-10-31 06:01:45,428 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:01:47,818 INFO scheduling health checks: 2019-10-31 06:01:47,837 INFO scheduling jobs: 2019-10-31 06:01:47,838 DEBUG - qemu 2019-10-31 06:01:47,901 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:01:47,901 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:01:47,906 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:01:49,043 INFO [INIT] Dropping privileges 2019-10-31 06:01:49,044 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:01:49,044 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:01:49,046 INFO [INIT] Marking all workers as offline 2019-10-31 06:01:49,106 INFO [INIT] LAVA master has started. 2019-10-31 06:01:49,107 INFO [INIT] Using protocol version 3 2019-10-31 06:01:52,187 DEBUG lava-logs => PING(20) 2019-10-31 06:02:05,448 DEBUG Dhanu => PING(20) 2019-10-31 06:02:05,449 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:02:09,111 INFO scheduling health checks: 2019-10-31 06:02:09,150 INFO scheduling jobs: 2019-10-31 06:02:09,154 DEBUG - qemu 2019-10-31 06:02:09,228 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:02:09,228 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:02:09,233 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:02:10,343 INFO [INIT] Dropping privileges 2019-10-31 06:02:10,343 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:02:10,343 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:02:10,344 INFO [INIT] Marking all workers as offline 2019-10-31 06:02:10,385 INFO [INIT] LAVA master has started. 2019-10-31 06:02:10,386 INFO [INIT] Using protocol version 3 2019-10-31 06:02:12,202 DEBUG lava-logs => PING(20) 2019-10-31 06:02:25,466 DEBUG Dhanu => PING(20) 2019-10-31 06:02:25,466 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:02:30,391 INFO scheduling health checks: 2019-10-31 06:02:30,430 INFO scheduling jobs: 2019-10-31 06:02:30,432 DEBUG - qemu 2019-10-31 06:02:30,505 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:02:30,505 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:02:30,510 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:02:31,789 INFO [INIT] Dropping privileges 2019-10-31 06:02:31,789 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:02:31,790 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:02:31,791 INFO [INIT] Marking all workers as offline 2019-10-31 06:02:31,833 INFO [INIT] LAVA master has started. 2019-10-31 06:02:31,833 INFO [INIT] Using protocol version 3 2019-10-31 06:02:32,216 DEBUG lava-logs => PING(20) 2019-10-31 06:02:45,488 DEBUG Dhanu => PING(20) 2019-10-31 06:02:45,488 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:02:51,839 INFO scheduling health checks: 2019-10-31 06:02:51,876 INFO scheduling jobs: 2019-10-31 06:02:51,879 DEBUG - qemu 2019-10-31 06:02:51,951 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:02:51,951 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:02:51,961 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:02:53,035 INFO [INIT] Dropping privileges 2019-10-31 06:02:53,035 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:02:53,035 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:02:53,037 INFO [INIT] Marking all workers as offline 2019-10-31 06:02:53,068 INFO [INIT] LAVA master has started. 2019-10-31 06:02:53,069 INFO [INIT] Using protocol version 3 2019-10-31 06:02:53,172 DEBUG lava-logs => PING(20) 2019-10-31 06:03:05,506 DEBUG Dhanu => PING(20) 2019-10-31 06:03:05,506 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:03:13,077 INFO scheduling health checks: 2019-10-31 06:03:13,123 INFO scheduling jobs: 2019-10-31 06:03:13,126 DEBUG - qemu 2019-10-31 06:03:13,205 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:03:13,205 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:03:13,209 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:03:14,281 INFO [INIT] Dropping privileges 2019-10-31 06:03:14,282 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:03:14,282 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:03:14,283 INFO [INIT] Marking all workers as offline 2019-10-31 06:03:14,314 INFO [INIT] LAVA master has started. 2019-10-31 06:03:14,314 INFO [INIT] Using protocol version 3 2019-10-31 06:03:25,527 DEBUG Dhanu => PING(20) 2019-10-31 06:03:25,527 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:03:33,200 DEBUG lava-logs => PING(20) 2019-10-31 06:03:34,315 INFO scheduling health checks: 2019-10-31 06:03:34,336 INFO scheduling jobs: 2019-10-31 06:03:34,338 DEBUG - qemu 2019-10-31 06:03:34,409 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:03:34,410 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:03:34,414 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:03:35,528 INFO [INIT] Dropping privileges 2019-10-31 06:03:35,528 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:03:35,528 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:03:35,530 INFO [INIT] Marking all workers as offline 2019-10-31 06:03:35,572 INFO [INIT] LAVA master has started. 2019-10-31 06:03:35,572 INFO [INIT] Using protocol version 3 2019-10-31 06:03:45,544 DEBUG Dhanu => PING(20) 2019-10-31 06:03:45,545 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:03:53,212 DEBUG lava-logs => PING(20) 2019-10-31 06:03:55,575 INFO scheduling health checks: 2019-10-31 06:03:55,605 INFO scheduling jobs: 2019-10-31 06:03:55,607 DEBUG - qemu 2019-10-31 06:03:55,677 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:03:55,677 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:03:55,682 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:03:56,799 INFO [INIT] Dropping privileges 2019-10-31 06:03:56,799 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:03:56,799 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:03:56,801 INFO [INIT] Marking all workers as offline 2019-10-31 06:03:56,841 INFO [INIT] LAVA master has started. 2019-10-31 06:03:56,841 INFO [INIT] Using protocol version 3 2019-10-31 06:04:05,567 DEBUG Dhanu => PING(20) 2019-10-31 06:04:05,568 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:04:13,232 DEBUG lava-logs => PING(20) 2019-10-31 06:04:16,843 INFO scheduling health checks: 2019-10-31 06:04:16,907 INFO scheduling jobs: 2019-10-31 06:04:16,910 DEBUG - qemu 2019-10-31 06:04:16,982 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:04:16,983 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:04:16,994 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:04:18,288 INFO [INIT] Dropping privileges 2019-10-31 06:04:18,288 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:04:18,289 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:04:18,290 INFO [INIT] Marking all workers as offline 2019-10-31 06:04:18,354 INFO [INIT] LAVA master has started. 2019-10-31 06:04:18,355 INFO [INIT] Using protocol version 3 2019-10-31 06:04:25,588 DEBUG Dhanu => PING(20) 2019-10-31 06:04:25,589 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:04:33,250 DEBUG lava-logs => PING(20) 2019-10-31 06:04:38,360 INFO scheduling health checks: 2019-10-31 06:04:38,387 INFO scheduling jobs: 2019-10-31 06:04:38,389 DEBUG - qemu 2019-10-31 06:04:38,461 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:04:38,461 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:04:38,465 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:04:39,533 INFO [INIT] Dropping privileges 2019-10-31 06:04:39,533 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:04:39,533 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:04:39,535 INFO [INIT] Marking all workers as offline 2019-10-31 06:04:39,578 INFO [INIT] LAVA master has started. 2019-10-31 06:04:39,579 INFO [INIT] Using protocol version 3 2019-10-31 06:04:45,608 DEBUG Dhanu => PING(20) 2019-10-31 06:04:45,608 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:04:53,264 DEBUG lava-logs => PING(20) 2019-10-31 06:04:59,586 INFO scheduling health checks: 2019-10-31 06:04:59,611 INFO scheduling jobs: 2019-10-31 06:04:59,612 DEBUG - qemu 2019-10-31 06:04:59,684 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:04:59,684 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:04:59,689 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:05:00,779 INFO [INIT] Dropping privileges 2019-10-31 06:05:00,780 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:05:00,780 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:05:00,781 INFO [INIT] Marking all workers as offline 2019-10-31 06:05:00,813 INFO [INIT] LAVA master has started. 2019-10-31 06:05:00,813 INFO [INIT] Using protocol version 3 2019-10-31 06:05:05,629 DEBUG Dhanu => PING(20) 2019-10-31 06:05:05,630 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:05:13,278 DEBUG lava-logs => PING(20) 2019-10-31 06:05:20,821 INFO scheduling health checks: 2019-10-31 06:05:20,880 INFO scheduling jobs: 2019-10-31 06:05:20,883 DEBUG - qemu 2019-10-31 06:05:20,957 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:05:20,958 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:05:20,962 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:05:22,064 INFO [INIT] Dropping privileges 2019-10-31 06:05:22,064 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:05:22,064 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:05:22,065 INFO [INIT] Marking all workers as offline 2019-10-31 06:05:22,127 INFO [INIT] LAVA master has started. 2019-10-31 06:05:22,127 INFO [INIT] Using protocol version 3 2019-10-31 06:05:25,652 DEBUG Dhanu => PING(20) 2019-10-31 06:05:25,653 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:05:33,284 DEBUG lava-logs => PING(20) 2019-10-31 06:05:42,136 INFO scheduling health checks: 2019-10-31 06:05:42,159 INFO scheduling jobs: 2019-10-31 06:05:42,161 DEBUG - qemu 2019-10-31 06:05:42,225 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:05:42,225 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:05:42,229 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:05:43,321 INFO [INIT] Dropping privileges 2019-10-31 06:05:43,321 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:05:43,321 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:05:43,323 INFO [INIT] Marking all workers as offline 2019-10-31 06:05:43,418 INFO [INIT] LAVA master has started. 2019-10-31 06:05:43,418 INFO [INIT] Using protocol version 3 2019-10-31 06:05:45,673 DEBUG Dhanu => PING(20) 2019-10-31 06:05:45,674 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:05:53,306 DEBUG lava-logs => PING(20) 2019-10-31 06:06:03,427 INFO scheduling health checks: 2019-10-31 06:06:03,462 INFO scheduling jobs: 2019-10-31 06:06:03,464 DEBUG - qemu 2019-10-31 06:06:03,535 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:06:03,535 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:06:03,539 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:06:04,789 INFO [INIT] Dropping privileges 2019-10-31 06:06:04,789 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:06:04,789 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:06:04,791 INFO [INIT] Marking all workers as offline 2019-10-31 06:06:04,842 INFO [INIT] LAVA master has started. 2019-10-31 06:06:04,842 INFO [INIT] Using protocol version 3 2019-10-31 06:06:05,694 DEBUG Dhanu => PING(20) 2019-10-31 06:06:05,694 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:06:13,328 DEBUG lava-logs => PING(20) 2019-10-31 06:06:24,854 INFO scheduling health checks: 2019-10-31 06:06:24,897 INFO scheduling jobs: 2019-10-31 06:06:24,899 DEBUG - qemu 2019-10-31 06:06:24,972 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:06:24,972 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:06:24,976 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:06:26,035 INFO [INIT] Dropping privileges 2019-10-31 06:06:26,036 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:06:26,036 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:06:26,037 INFO [INIT] Marking all workers as offline 2019-10-31 06:06:26,077 INFO [INIT] LAVA master has started. 2019-10-31 06:06:26,077 INFO [INIT] Using protocol version 3 2019-10-31 06:06:26,092 DEBUG Dhanu => PING(20) 2019-10-31 06:06:26,092 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:06:33,349 DEBUG lava-logs => PING(20) 2019-10-31 06:06:46,083 INFO scheduling health checks: 2019-10-31 06:06:46,110 INFO scheduling jobs: 2019-10-31 06:06:46,112 DEBUG - qemu 2019-10-31 06:06:46,187 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:06:46,187 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:06:46,191 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:06:47,293 INFO [INIT] Dropping privileges 2019-10-31 06:06:47,293 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:06:47,294 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:06:47,295 INFO [INIT] Marking all workers as offline 2019-10-31 06:06:47,335 INFO [INIT] LAVA master has started. 2019-10-31 06:06:47,336 INFO [INIT] Using protocol version 3 2019-10-31 06:06:53,359 DEBUG lava-logs => PING(20) 2019-10-31 06:07:06,132 DEBUG Dhanu => PING(20) 2019-10-31 06:07:06,133 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:07:07,337 INFO scheduling health checks: 2019-10-31 06:07:07,357 INFO scheduling jobs: 2019-10-31 06:07:07,358 DEBUG - qemu 2019-10-31 06:07:07,428 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:07:07,428 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:07:07,433 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:07:08,538 INFO [INIT] Dropping privileges 2019-10-31 06:07:08,538 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:07:08,539 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:07:08,540 INFO [INIT] Marking all workers as offline 2019-10-31 06:07:08,582 INFO [INIT] LAVA master has started. 2019-10-31 06:07:08,582 INFO [INIT] Using protocol version 3 2019-10-31 06:07:13,377 DEBUG lava-logs => PING(20) 2019-10-31 06:07:26,152 DEBUG Dhanu => PING(20) 2019-10-31 06:07:26,153 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:07:28,585 INFO scheduling health checks: 2019-10-31 06:07:28,614 INFO scheduling jobs: 2019-10-31 06:07:28,616 DEBUG - qemu 2019-10-31 06:07:28,691 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:07:28,691 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:07:28,695 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:07:29,819 INFO [INIT] Dropping privileges 2019-10-31 06:07:29,820 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:07:29,820 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:07:29,821 INFO [INIT] Marking all workers as offline 2019-10-31 06:07:29,884 INFO [INIT] LAVA master has started. 2019-10-31 06:07:29,885 INFO [INIT] Using protocol version 3 2019-10-31 06:07:33,393 DEBUG lava-logs => PING(20) 2019-10-31 06:07:46,170 DEBUG Dhanu => PING(20) 2019-10-31 06:07:46,170 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:07:49,888 INFO scheduling health checks: 2019-10-31 06:07:49,928 INFO scheduling jobs: 2019-10-31 06:07:49,931 DEBUG - qemu 2019-10-31 06:07:50,006 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:07:50,006 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:07:50,011 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:07:51,284 INFO [INIT] Dropping privileges 2019-10-31 06:07:51,284 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:07:51,284 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:07:51,286 INFO [INIT] Marking all workers as offline 2019-10-31 06:07:51,319 INFO [INIT] LAVA master has started. 2019-10-31 06:07:51,319 INFO [INIT] Using protocol version 3 2019-10-31 06:07:53,408 DEBUG lava-logs => PING(20) 2019-10-31 06:08:06,192 DEBUG Dhanu => PING(20) 2019-10-31 06:08:06,193 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:08:11,325 INFO scheduling health checks: 2019-10-31 06:08:11,363 INFO scheduling jobs: 2019-10-31 06:08:11,366 DEBUG - qemu 2019-10-31 06:08:11,436 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:08:11,436 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:08:11,441 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:08:12,547 INFO [INIT] Dropping privileges 2019-10-31 06:08:12,547 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:08:12,548 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:08:12,549 INFO [INIT] Marking all workers as offline 2019-10-31 06:08:12,599 INFO [INIT] LAVA master has started. 2019-10-31 06:08:12,599 INFO [INIT] Using protocol version 3 2019-10-31 06:08:13,422 DEBUG lava-logs => PING(20) 2019-10-31 06:08:26,211 DEBUG Dhanu => PING(20) 2019-10-31 06:08:26,212 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:08:32,606 INFO scheduling health checks: 2019-10-31 06:08:32,632 INFO scheduling jobs: 2019-10-31 06:08:32,633 DEBUG - qemu 2019-10-31 06:08:32,705 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:08:32,705 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:08:32,710 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:08:33,779 INFO [INIT] Dropping privileges 2019-10-31 06:08:33,779 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:08:33,779 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:08:33,781 INFO [INIT] Marking all workers as offline 2019-10-31 06:08:33,823 INFO [INIT] LAVA master has started. 2019-10-31 06:08:33,824 INFO [INIT] Using protocol version 3 2019-10-31 06:08:33,950 DEBUG lava-logs => PING(20) 2019-10-31 06:08:46,233 DEBUG Dhanu => PING(20) 2019-10-31 06:08:46,233 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:08:53,832 INFO scheduling health checks: 2019-10-31 06:08:53,867 INFO scheduling jobs: 2019-10-31 06:08:53,869 DEBUG - qemu 2019-10-31 06:08:53,941 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:08:53,941 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:08:53,946 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:08:55,036 INFO [INIT] Dropping privileges 2019-10-31 06:08:55,036 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:08:55,036 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:08:55,038 INFO [INIT] Marking all workers as offline 2019-10-31 06:08:55,081 INFO [INIT] LAVA master has started. 2019-10-31 06:08:55,081 INFO [INIT] Using protocol version 3 2019-10-31 06:08:55,195 DEBUG lava-logs => PING(20) 2019-10-31 06:09:06,251 DEBUG Dhanu => PING(20) 2019-10-31 06:09:06,252 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:09:15,090 INFO scheduling health checks: 2019-10-31 06:09:15,125 INFO scheduling jobs: 2019-10-31 06:09:15,127 DEBUG - qemu 2019-10-31 06:09:15,200 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:09:15,200 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:09:15,205 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:09:16,307 INFO [INIT] Dropping privileges 2019-10-31 06:09:16,307 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:09:16,307 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:09:16,309 INFO [INIT] Marking all workers as offline 2019-10-31 06:09:16,372 INFO [INIT] LAVA master has started. 2019-10-31 06:09:16,372 INFO [INIT] Using protocol version 3 2019-10-31 06:09:16,383 DEBUG lava-logs => PING(20) 2019-10-31 06:09:26,275 DEBUG Dhanu => PING(20) 2019-10-31 06:09:26,275 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:09:36,379 INFO scheduling health checks: 2019-10-31 06:09:36,404 INFO scheduling jobs: 2019-10-31 06:09:36,406 DEBUG - qemu 2019-10-31 06:09:36,472 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:09:36,472 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:09:36,477 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:09:37,527 INFO [INIT] Dropping privileges 2019-10-31 06:09:37,528 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:09:37,528 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:09:37,529 INFO [INIT] Marking all workers as offline 2019-10-31 06:09:37,562 INFO [INIT] LAVA master has started. 2019-10-31 06:09:37,563 INFO [INIT] Using protocol version 3 2019-10-31 06:09:46,296 DEBUG Dhanu => PING(20) 2019-10-31 06:09:46,297 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:09:56,416 DEBUG lava-logs => PING(20) 2019-10-31 06:09:57,564 INFO scheduling health checks: 2019-10-31 06:09:57,584 INFO scheduling jobs: 2019-10-31 06:09:57,586 DEBUG - qemu 2019-10-31 06:09:57,654 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:09:57,654 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:09:57,658 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:09:58,831 INFO [INIT] Dropping privileges 2019-10-31 06:09:58,831 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:09:58,831 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:09:58,833 INFO [INIT] Marking all workers as offline 2019-10-31 06:09:58,865 INFO [INIT] LAVA master has started. 2019-10-31 06:09:58,865 INFO [INIT] Using protocol version 3 2019-10-31 06:10:06,318 DEBUG Dhanu => PING(20) 2019-10-31 06:10:06,318 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:10:16,436 DEBUG lava-logs => PING(20) 2019-10-31 06:10:18,867 INFO scheduling health checks: 2019-10-31 06:10:18,886 INFO scheduling jobs: 2019-10-31 06:10:18,887 DEBUG - qemu 2019-10-31 06:10:18,953 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:10:18,953 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:10:18,958 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:10:20,030 INFO [INIT] Dropping privileges 2019-10-31 06:10:20,030 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:10:20,030 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:10:20,032 INFO [INIT] Marking all workers as offline 2019-10-31 06:10:20,066 INFO [INIT] LAVA master has started. 2019-10-31 06:10:20,066 INFO [INIT] Using protocol version 3 2019-10-31 06:10:26,339 DEBUG Dhanu => PING(20) 2019-10-31 06:10:26,339 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:10:36,456 DEBUG lava-logs => PING(20) 2019-10-31 06:10:40,071 INFO scheduling health checks: 2019-10-31 06:10:40,099 INFO scheduling jobs: 2019-10-31 06:10:40,101 DEBUG - qemu 2019-10-31 06:10:40,167 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:10:40,168 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:10:40,177 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:10:41,308 INFO [INIT] Dropping privileges 2019-10-31 06:10:41,308 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:10:41,308 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:10:41,310 INFO [INIT] Marking all workers as offline 2019-10-31 06:10:41,390 INFO [INIT] LAVA master has started. 2019-10-31 06:10:41,390 INFO [INIT] Using protocol version 3 2019-10-31 06:10:46,362 DEBUG Dhanu => PING(20) 2019-10-31 06:10:46,362 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:10:56,474 DEBUG lava-logs => PING(20) 2019-10-31 06:11:01,395 INFO scheduling health checks: 2019-10-31 06:11:01,424 INFO scheduling jobs: 2019-10-31 06:11:01,426 DEBUG - qemu 2019-10-31 06:11:01,499 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:11:01,499 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:11:01,511 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:11:02,795 INFO [INIT] Dropping privileges 2019-10-31 06:11:02,795 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:11:02,795 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:11:02,797 INFO [INIT] Marking all workers as offline 2019-10-31 06:11:02,838 INFO [INIT] LAVA master has started. 2019-10-31 06:11:02,838 INFO [INIT] Using protocol version 3 2019-10-31 06:11:06,384 DEBUG Dhanu => PING(20) 2019-10-31 06:11:06,385 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:11:16,490 DEBUG lava-logs => PING(20) 2019-10-31 06:11:22,845 INFO scheduling health checks: 2019-10-31 06:11:22,882 INFO scheduling jobs: 2019-10-31 06:11:22,885 DEBUG - qemu 2019-10-31 06:11:22,960 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:11:22,961 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:11:22,975 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:11:24,025 INFO [INIT] Dropping privileges 2019-10-31 06:11:24,025 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:11:24,025 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:11:24,027 INFO [INIT] Marking all workers as offline 2019-10-31 06:11:24,062 INFO [INIT] LAVA master has started. 2019-10-31 06:11:24,062 INFO [INIT] Using protocol version 3 2019-10-31 06:11:26,403 DEBUG Dhanu => PING(20) 2019-10-31 06:11:26,404 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:11:36,506 DEBUG lava-logs => PING(20) 2019-10-31 06:11:44,070 INFO scheduling health checks: 2019-10-31 06:11:44,117 INFO scheduling jobs: 2019-10-31 06:11:44,119 DEBUG - qemu 2019-10-31 06:11:44,187 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:11:44,188 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:11:44,192 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:11:45,274 INFO [INIT] Dropping privileges 2019-10-31 06:11:45,275 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:11:45,275 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:11:45,276 INFO [INIT] Marking all workers as offline 2019-10-31 06:11:45,331 INFO [INIT] LAVA master has started. 2019-10-31 06:11:45,331 INFO [INIT] Using protocol version 3 2019-10-31 06:11:46,421 DEBUG Dhanu => PING(20) 2019-10-31 06:11:46,422 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:11:56,516 DEBUG lava-logs => PING(20) 2019-10-31 06:12:05,339 INFO scheduling health checks: 2019-10-31 06:12:05,375 INFO scheduling jobs: 2019-10-31 06:12:05,377 DEBUG - qemu 2019-10-31 06:12:05,453 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:12:05,453 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:12:05,458 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:12:06,533 INFO [INIT] Dropping privileges 2019-10-31 06:12:06,533 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:12:06,533 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:12:06,535 INFO [INIT] Marking all workers as offline 2019-10-31 06:12:06,566 INFO [INIT] LAVA master has started. 2019-10-31 06:12:06,566 INFO [INIT] Using protocol version 3 2019-10-31 06:12:06,665 DEBUG Dhanu => PING(20) 2019-10-31 06:12:06,665 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:12:16,539 DEBUG lava-logs => PING(20) 2019-10-31 06:12:26,577 INFO scheduling health checks: 2019-10-31 06:12:26,621 INFO scheduling jobs: 2019-10-31 06:12:26,623 DEBUG - qemu 2019-10-31 06:12:26,695 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:12:26,695 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:12:26,700 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:12:27,890 INFO [INIT] Dropping privileges 2019-10-31 06:12:27,890 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:12:27,890 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:12:27,892 INFO [INIT] Marking all workers as offline 2019-10-31 06:12:27,923 INFO [INIT] LAVA master has started. 2019-10-31 06:12:27,923 INFO [INIT] Using protocol version 3 2019-10-31 06:12:36,560 DEBUG lava-logs => PING(20) 2019-10-31 06:12:46,709 DEBUG Dhanu => PING(20) 2019-10-31 06:12:46,710 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:12:47,925 INFO scheduling health checks: 2019-10-31 06:12:47,956 INFO scheduling jobs: 2019-10-31 06:12:47,958 DEBUG - qemu 2019-10-31 06:12:48,037 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:12:48,037 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:12:48,042 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:12:49,399 INFO [INIT] Dropping privileges 2019-10-31 06:12:49,400 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:12:49,400 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:12:49,401 INFO [INIT] Marking all workers as offline 2019-10-31 06:12:49,458 INFO [INIT] LAVA master has started. 2019-10-31 06:12:49,458 INFO [INIT] Using protocol version 3 2019-10-31 06:12:56,574 DEBUG lava-logs => PING(20) 2019-10-31 06:13:06,727 DEBUG Dhanu => PING(20) 2019-10-31 06:13:06,728 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:13:09,459 INFO scheduling health checks: 2019-10-31 06:13:09,503 INFO scheduling jobs: 2019-10-31 06:13:09,506 DEBUG - qemu 2019-10-31 06:13:09,593 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:13:09,594 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:13:09,599 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:13:11,147 INFO [INIT] Dropping privileges 2019-10-31 06:13:11,147 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:13:11,147 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:13:11,151 INFO [INIT] Marking all workers as offline 2019-10-31 06:13:11,194 INFO [INIT] LAVA master has started. 2019-10-31 06:13:11,195 INFO [INIT] Using protocol version 3 2019-10-31 06:13:16,590 DEBUG lava-logs => PING(20) 2019-10-31 06:13:26,744 DEBUG Dhanu => PING(20) 2019-10-31 06:13:26,745 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:13:31,199 INFO scheduling health checks: 2019-10-31 06:13:31,261 INFO scheduling jobs: 2019-10-31 06:13:31,265 DEBUG - qemu 2019-10-31 06:13:31,346 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:13:31,346 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:13:31,351 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:13:32,567 INFO [INIT] Dropping privileges 2019-10-31 06:13:32,568 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:13:32,568 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:13:32,569 INFO [INIT] Marking all workers as offline 2019-10-31 06:13:32,619 INFO [INIT] LAVA master has started. 2019-10-31 06:13:32,619 INFO [INIT] Using protocol version 3 2019-10-31 06:13:36,606 DEBUG lava-logs => PING(20) 2019-10-31 06:13:46,760 DEBUG Dhanu => PING(20) 2019-10-31 06:13:46,761 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:13:52,626 INFO scheduling health checks: 2019-10-31 06:13:52,663 INFO scheduling jobs: 2019-10-31 06:13:52,665 DEBUG - qemu 2019-10-31 06:13:52,738 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:13:52,738 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:13:52,742 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:13:54,056 INFO [INIT] Dropping privileges 2019-10-31 06:13:54,057 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:13:54,057 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:13:54,058 INFO [INIT] Marking all workers as offline 2019-10-31 06:13:54,121 INFO [INIT] LAVA master has started. 2019-10-31 06:13:54,121 INFO [INIT] Using protocol version 3 2019-10-31 06:13:56,619 DEBUG lava-logs => PING(20) 2019-10-31 06:14:06,782 DEBUG Dhanu => PING(20) 2019-10-31 06:14:06,782 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:14:14,129 INFO scheduling health checks: 2019-10-31 06:14:14,165 INFO scheduling jobs: 2019-10-31 06:14:14,168 DEBUG - qemu 2019-10-31 06:14:14,238 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:14:14,238 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:14:14,242 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:14:15,346 INFO [INIT] Dropping privileges 2019-10-31 06:14:15,347 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:14:15,347 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:14:15,348 INFO [INIT] Marking all workers as offline 2019-10-31 06:14:15,377 INFO [INIT] LAVA master has started. 2019-10-31 06:14:15,377 INFO [INIT] Using protocol version 3 2019-10-31 06:14:16,634 DEBUG lava-logs => PING(20) 2019-10-31 06:14:26,802 DEBUG Dhanu => PING(20) 2019-10-31 06:14:26,802 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:14:35,386 INFO scheduling health checks: 2019-10-31 06:14:35,422 INFO scheduling jobs: 2019-10-31 06:14:35,424 DEBUG - qemu 2019-10-31 06:14:35,496 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:14:35,496 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:14:35,501 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:14:36,794 INFO [INIT] Dropping privileges 2019-10-31 06:14:36,794 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:14:36,794 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:14:36,796 INFO [INIT] Marking all workers as offline 2019-10-31 06:14:36,847 INFO [INIT] LAVA master has started. 2019-10-31 06:14:36,847 INFO [INIT] Using protocol version 3 2019-10-31 06:14:36,979 DEBUG lava-logs => PING(20) 2019-10-31 06:14:46,824 DEBUG Dhanu => PING(20) 2019-10-31 06:14:46,825 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:14:56,855 INFO scheduling health checks: 2019-10-31 06:14:56,902 INFO scheduling jobs: 2019-10-31 06:14:56,905 DEBUG - qemu 2019-10-31 06:14:56,986 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:14:56,986 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:14:56,992 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:14:58,386 INFO [INIT] Dropping privileges 2019-10-31 06:14:58,386 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:14:58,386 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:14:58,388 INFO [INIT] Marking all workers as offline 2019-10-31 06:14:58,429 INFO [INIT] LAVA master has started. 2019-10-31 06:14:58,429 INFO [INIT] Using protocol version 3 2019-10-31 06:14:58,503 DEBUG lava-logs => PING(20) 2019-10-31 06:15:06,839 DEBUG Dhanu => PING(20) 2019-10-31 06:15:06,840 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:15:18,439 INFO scheduling health checks: 2019-10-31 06:15:18,460 INFO scheduling jobs: 2019-10-31 06:15:18,461 DEBUG - qemu 2019-10-31 06:15:18,535 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:15:18,535 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:15:18,540 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:15:19,813 INFO [INIT] Dropping privileges 2019-10-31 06:15:19,814 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:15:19,838 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:15:19,840 INFO [INIT] Marking all workers as offline 2019-10-31 06:15:19,919 INFO [INIT] LAVA master has started. 2019-10-31 06:15:19,920 INFO [INIT] Using protocol version 3 2019-10-31 06:15:26,858 DEBUG Dhanu => PING(20) 2019-10-31 06:15:26,858 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:15:38,545 DEBUG lava-logs => PING(20) 2019-10-31 06:15:39,921 INFO scheduling health checks: 2019-10-31 06:15:39,951 INFO scheduling jobs: 2019-10-31 06:15:39,953 DEBUG - qemu 2019-10-31 06:15:40,027 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:15:40,027 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:15:40,031 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:15:41,288 INFO [INIT] Dropping privileges 2019-10-31 06:15:41,289 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:15:41,289 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:15:41,290 INFO [INIT] Marking all workers as offline 2019-10-31 06:15:41,332 INFO [INIT] LAVA master has started. 2019-10-31 06:15:41,332 INFO [INIT] Using protocol version 3 2019-10-31 06:15:46,874 DEBUG Dhanu => PING(20) 2019-10-31 06:15:46,875 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:15:58,560 DEBUG lava-logs => PING(20) 2019-10-31 06:16:01,335 INFO scheduling health checks: 2019-10-31 06:16:01,375 INFO scheduling jobs: 2019-10-31 06:16:01,377 DEBUG - qemu 2019-10-31 06:16:01,446 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:16:01,446 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:16:01,450 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:16:02,532 INFO [INIT] Dropping privileges 2019-10-31 06:16:02,532 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:16:02,532 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:16:02,534 INFO [INIT] Marking all workers as offline 2019-10-31 06:16:02,566 INFO [INIT] LAVA master has started. 2019-10-31 06:16:02,567 INFO [INIT] Using protocol version 3 2019-10-31 06:16:06,893 DEBUG Dhanu => PING(20) 2019-10-31 06:16:06,894 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:16:18,577 DEBUG lava-logs => PING(20) 2019-10-31 06:16:22,571 INFO scheduling health checks: 2019-10-31 06:16:22,599 INFO scheduling jobs: 2019-10-31 06:16:22,601 DEBUG - qemu 2019-10-31 06:16:22,666 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:16:22,667 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:16:22,671 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:16:23,804 INFO [INIT] Dropping privileges 2019-10-31 06:16:23,805 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:16:23,805 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:16:23,806 INFO [INIT] Marking all workers as offline 2019-10-31 06:16:23,845 INFO [INIT] LAVA master has started. 2019-10-31 06:16:23,846 INFO [INIT] Using protocol version 3 2019-10-31 06:16:26,915 DEBUG Dhanu => PING(20) 2019-10-31 06:16:26,915 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:16:38,594 DEBUG lava-logs => PING(20) 2019-10-31 06:16:43,851 INFO scheduling health checks: 2019-10-31 06:16:43,902 INFO scheduling jobs: 2019-10-31 06:16:43,905 DEBUG - qemu 2019-10-31 06:16:43,981 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:16:43,981 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:16:43,992 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:16:45,281 INFO [INIT] Dropping privileges 2019-10-31 06:16:45,281 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:16:45,281 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:16:45,283 INFO [INIT] Marking all workers as offline 2019-10-31 06:16:45,315 INFO [INIT] LAVA master has started. 2019-10-31 06:16:45,315 INFO [INIT] Using protocol version 3 2019-10-31 06:16:46,932 DEBUG Dhanu => PING(20) 2019-10-31 06:16:46,932 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:16:58,608 DEBUG lava-logs => PING(20) 2019-10-31 06:17:05,323 INFO scheduling health checks: 2019-10-31 06:17:05,359 INFO scheduling jobs: 2019-10-31 06:17:05,363 DEBUG - qemu 2019-10-31 06:17:05,437 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:17:05,437 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:17:05,442 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:17:06,527 INFO [INIT] Dropping privileges 2019-10-31 06:17:06,528 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:17:06,528 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:17:06,529 INFO [INIT] Marking all workers as offline 2019-10-31 06:17:06,562 INFO [INIT] LAVA master has started. 2019-10-31 06:17:06,562 INFO [INIT] Using protocol version 3 2019-10-31 06:17:06,952 DEBUG Dhanu => PING(20) 2019-10-31 06:17:06,952 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:17:18,623 DEBUG lava-logs => PING(20) 2019-10-31 06:17:26,570 INFO scheduling health checks: 2019-10-31 06:17:26,606 INFO scheduling jobs: 2019-10-31 06:17:26,608 DEBUG - qemu 2019-10-31 06:17:26,684 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:17:26,684 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:17:26,688 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:17:27,774 INFO [INIT] Dropping privileges 2019-10-31 06:17:27,774 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:17:27,774 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:17:27,776 INFO [INIT] Marking all workers as offline 2019-10-31 06:17:27,809 INFO [INIT] LAVA master has started. 2019-10-31 06:17:27,809 INFO [INIT] Using protocol version 3 2019-10-31 06:17:27,842 DEBUG Dhanu => PING(20) 2019-10-31 06:17:27,842 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:17:38,637 DEBUG lava-logs => PING(20) 2019-10-31 06:17:47,819 INFO scheduling health checks: 2019-10-31 06:17:47,852 INFO scheduling jobs: 2019-10-31 06:17:47,855 DEBUG - qemu 2019-10-31 06:17:47,930 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:17:47,930 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:17:47,935 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:17:49,036 INFO [INIT] Dropping privileges 2019-10-31 06:17:49,037 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:17:49,037 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:17:49,038 INFO [INIT] Marking all workers as offline 2019-10-31 06:17:49,076 INFO [INIT] LAVA master has started. 2019-10-31 06:17:49,076 INFO [INIT] Using protocol version 3 2019-10-31 06:17:58,660 DEBUG lava-logs => PING(20) 2019-10-31 06:18:07,884 DEBUG Dhanu => PING(20) 2019-10-31 06:18:07,885 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:18:09,077 INFO scheduling health checks: 2019-10-31 06:18:09,099 INFO scheduling jobs: 2019-10-31 06:18:09,100 DEBUG - qemu 2019-10-31 06:18:09,165 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:18:09,165 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:18:09,169 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:18:10,285 INFO [INIT] Dropping privileges 2019-10-31 06:18:10,285 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:18:10,285 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:18:10,287 INFO [INIT] Marking all workers as offline 2019-10-31 06:18:10,323 INFO [INIT] LAVA master has started. 2019-10-31 06:18:10,324 INFO [INIT] Using protocol version 3 2019-10-31 06:18:18,681 DEBUG lava-logs => PING(20) 2019-10-31 06:18:27,900 DEBUG Dhanu => PING(20) 2019-10-31 06:18:27,901 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:18:30,326 INFO scheduling health checks: 2019-10-31 06:18:30,345 INFO scheduling jobs: 2019-10-31 06:18:30,346 DEBUG - qemu 2019-10-31 06:18:30,416 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:18:30,417 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:18:30,424 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:18:31,537 INFO [INIT] Dropping privileges 2019-10-31 06:18:31,538 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:18:31,538 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:18:31,539 INFO [INIT] Marking all workers as offline 2019-10-31 06:18:31,569 INFO [INIT] LAVA master has started. 2019-10-31 06:18:31,569 INFO [INIT] Using protocol version 3 2019-10-31 06:18:38,701 DEBUG lava-logs => PING(20) 2019-10-31 06:18:47,921 DEBUG Dhanu => PING(20) 2019-10-31 06:18:47,921 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:18:51,574 INFO scheduling health checks: 2019-10-31 06:18:51,614 INFO scheduling jobs: 2019-10-31 06:18:51,616 DEBUG - qemu 2019-10-31 06:18:51,690 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:18:51,690 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:18:51,695 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:18:52,784 INFO [INIT] Dropping privileges 2019-10-31 06:18:52,785 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:18:52,785 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:18:52,787 INFO [INIT] Marking all workers as offline 2019-10-31 06:18:52,844 INFO [INIT] LAVA master has started. 2019-10-31 06:18:52,844 INFO [INIT] Using protocol version 3 2019-10-31 06:18:58,719 DEBUG lava-logs => PING(20) 2019-10-31 06:19:07,936 DEBUG Dhanu => PING(20) 2019-10-31 06:19:07,937 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:19:12,847 INFO scheduling health checks: 2019-10-31 06:19:12,883 INFO scheduling jobs: 2019-10-31 06:19:12,886 DEBUG - qemu 2019-10-31 06:19:12,961 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:19:12,961 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:19:12,965 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:19:14,030 INFO [INIT] Dropping privileges 2019-10-31 06:19:14,031 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:19:14,031 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:19:14,032 INFO [INIT] Marking all workers as offline 2019-10-31 06:19:14,062 INFO [INIT] LAVA master has started. 2019-10-31 06:19:14,062 INFO [INIT] Using protocol version 3 2019-10-31 06:19:18,737 DEBUG lava-logs => PING(20) 2019-10-31 06:19:27,956 DEBUG Dhanu => PING(20) 2019-10-31 06:19:27,956 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:19:34,069 INFO scheduling health checks: 2019-10-31 06:19:34,107 INFO scheduling jobs: 2019-10-31 06:19:34,111 DEBUG - qemu 2019-10-31 06:19:34,185 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:19:34,185 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:19:34,190 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:19:35,285 INFO [INIT] Dropping privileges 2019-10-31 06:19:35,285 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:19:35,285 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:19:35,287 INFO [INIT] Marking all workers as offline 2019-10-31 06:19:35,320 INFO [INIT] LAVA master has started. 2019-10-31 06:19:35,321 INFO [INIT] Using protocol version 3 2019-10-31 06:19:38,754 DEBUG lava-logs => PING(20) 2019-10-31 06:19:47,977 DEBUG Dhanu => PING(20) 2019-10-31 06:19:47,978 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:19:55,329 INFO scheduling health checks: 2019-10-31 06:19:55,387 INFO scheduling jobs: 2019-10-31 06:19:55,390 DEBUG - qemu 2019-10-31 06:19:55,464 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:19:55,464 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:19:55,469 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:19:56,533 INFO [INIT] Dropping privileges 2019-10-31 06:19:56,533 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:19:56,533 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:19:56,534 INFO [INIT] Marking all workers as offline 2019-10-31 06:19:56,590 INFO [INIT] LAVA master has started. 2019-10-31 06:19:56,590 INFO [INIT] Using protocol version 3 2019-10-31 06:19:58,767 DEBUG lava-logs => PING(20) 2019-10-31 06:20:07,998 DEBUG Dhanu => PING(20) 2019-10-31 06:20:07,999 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:20:16,599 INFO scheduling health checks: 2019-10-31 06:20:16,633 INFO scheduling jobs: 2019-10-31 06:20:16,635 DEBUG - qemu 2019-10-31 06:20:16,701 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:20:16,702 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:20:16,706 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:20:17,778 INFO [INIT] Dropping privileges 2019-10-31 06:20:17,778 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:20:17,778 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:20:17,780 INFO [INIT] Marking all workers as offline 2019-10-31 06:20:17,813 INFO [INIT] LAVA master has started. 2019-10-31 06:20:17,813 INFO [INIT] Using protocol version 3 2019-10-31 06:20:18,781 DEBUG lava-logs => PING(20) 2019-10-31 06:20:28,018 DEBUG Dhanu => PING(20) 2019-10-31 06:20:28,019 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:20:37,823 INFO scheduling health checks: 2019-10-31 06:20:37,846 INFO scheduling jobs: 2019-10-31 06:20:37,848 DEBUG - qemu 2019-10-31 06:20:37,914 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:20:37,914 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:20:37,918 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:20:39,081 INFO [INIT] Dropping privileges 2019-10-31 06:20:39,082 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:20:39,082 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:20:39,083 INFO [INIT] Marking all workers as offline 2019-10-31 06:20:39,161 INFO [INIT] LAVA master has started. 2019-10-31 06:20:39,161 INFO [INIT] Using protocol version 3 2019-10-31 06:20:39,166 DEBUG lava-logs => PING(20) 2019-10-31 06:20:48,038 DEBUG Dhanu => PING(20) 2019-10-31 06:20:48,038 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:20:59,173 INFO scheduling health checks: 2019-10-31 06:20:59,193 INFO scheduling jobs: 2019-10-31 06:20:59,194 DEBUG - qemu 2019-10-31 06:20:59,265 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:20:59,265 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:20:59,270 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:21:00,337 INFO [INIT] Dropping privileges 2019-10-31 06:21:00,338 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:21:00,338 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:21:00,339 INFO [INIT] Marking all workers as offline 2019-10-31 06:21:00,373 INFO [INIT] LAVA master has started. 2019-10-31 06:21:00,373 INFO [INIT] Using protocol version 3 2019-10-31 06:21:08,059 DEBUG Dhanu => PING(20) 2019-10-31 06:21:08,059 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:21:19,208 DEBUG lava-logs => PING(20) 2019-10-31 06:21:20,375 INFO scheduling health checks: 2019-10-31 06:21:20,406 INFO scheduling jobs: 2019-10-31 06:21:20,408 DEBUG - qemu 2019-10-31 06:21:20,474 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:21:20,474 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:21:20,478 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:21:21,773 INFO [INIT] Dropping privileges 2019-10-31 06:21:21,773 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:21:21,774 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:21:21,775 INFO [INIT] Marking all workers as offline 2019-10-31 06:21:21,809 INFO [INIT] LAVA master has started. 2019-10-31 06:21:21,809 INFO [INIT] Using protocol version 3 2019-10-31 06:21:28,076 DEBUG Dhanu => PING(20) 2019-10-31 06:21:28,077 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:21:39,216 DEBUG lava-logs => PING(20) 2019-10-31 06:21:41,811 INFO scheduling health checks: 2019-10-31 06:21:41,842 INFO scheduling jobs: 2019-10-31 06:21:41,844 DEBUG - qemu 2019-10-31 06:21:41,915 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:21:41,916 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:21:41,920 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:21:43,045 INFO [INIT] Dropping privileges 2019-10-31 06:21:43,045 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:21:43,045 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:21:43,047 INFO [INIT] Marking all workers as offline 2019-10-31 06:21:43,133 INFO [INIT] LAVA master has started. 2019-10-31 06:21:43,134 INFO [INIT] Using protocol version 3 2019-10-31 06:21:48,098 DEBUG Dhanu => PING(20) 2019-10-31 06:21:48,098 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:21:59,235 DEBUG lava-logs => PING(20) 2019-10-31 06:22:03,135 INFO scheduling health checks: 2019-10-31 06:22:03,166 INFO scheduling jobs: 2019-10-31 06:22:03,170 DEBUG - qemu 2019-10-31 06:22:03,270 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:22:03,270 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:22:03,277 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:22:04,628 INFO [INIT] Dropping privileges 2019-10-31 06:22:04,629 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:22:04,629 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:22:04,632 INFO [INIT] Marking all workers as offline 2019-10-31 06:22:04,656 INFO [INIT] LAVA master has started. 2019-10-31 06:22:04,656 INFO [INIT] Using protocol version 3 2019-10-31 06:22:08,120 DEBUG Dhanu => PING(20) 2019-10-31 06:22:08,120 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:22:19,248 DEBUG lava-logs => PING(20) 2019-10-31 06:22:24,658 INFO scheduling health checks: 2019-10-31 06:22:24,701 INFO scheduling jobs: 2019-10-31 06:22:24,704 DEBUG - qemu 2019-10-31 06:22:24,781 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:22:24,781 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:22:24,786 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:22:26,050 INFO [INIT] Dropping privileges 2019-10-31 06:22:26,051 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:22:26,051 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:22:26,052 INFO [INIT] Marking all workers as offline 2019-10-31 06:22:26,092 INFO [INIT] LAVA master has started. 2019-10-31 06:22:26,092 INFO [INIT] Using protocol version 3 2019-10-31 06:22:28,132 DEBUG Dhanu => PING(20) 2019-10-31 06:22:28,133 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:22:39,260 DEBUG lava-logs => PING(20) 2019-10-31 06:22:46,099 INFO scheduling health checks: 2019-10-31 06:22:46,137 INFO scheduling jobs: 2019-10-31 06:22:46,140 DEBUG - qemu 2019-10-31 06:22:46,208 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:22:46,208 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:22:46,213 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:22:47,296 INFO [INIT] Dropping privileges 2019-10-31 06:22:47,296 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:22:47,296 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:22:47,298 INFO [INIT] Marking all workers as offline 2019-10-31 06:22:47,327 INFO [INIT] LAVA master has started. 2019-10-31 06:22:47,328 INFO [INIT] Using protocol version 3 2019-10-31 06:22:48,149 DEBUG Dhanu => PING(20) 2019-10-31 06:22:48,150 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:22:59,275 DEBUG lava-logs => PING(20) 2019-10-31 06:23:07,334 INFO scheduling health checks: 2019-10-31 06:23:07,372 INFO scheduling jobs: 2019-10-31 06:23:07,374 DEBUG - qemu 2019-10-31 06:23:07,446 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:23:07,446 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:23:07,451 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:23:08,546 INFO [INIT] Dropping privileges 2019-10-31 06:23:08,546 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:23:08,546 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:23:08,548 INFO [INIT] Marking all workers as offline 2019-10-31 06:23:08,585 INFO [INIT] LAVA master has started. 2019-10-31 06:23:08,585 INFO [INIT] Using protocol version 3 2019-10-31 06:23:08,588 DEBUG Dhanu => PING(20) 2019-10-31 06:23:08,588 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:23:19,287 DEBUG lava-logs => PING(20) 2019-10-31 06:23:28,591 INFO scheduling health checks: 2019-10-31 06:23:28,630 INFO scheduling jobs: 2019-10-31 06:23:28,632 DEBUG - qemu 2019-10-31 06:23:28,705 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:23:28,705 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:23:28,709 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:23:29,777 INFO [INIT] Dropping privileges 2019-10-31 06:23:29,778 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:23:29,778 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:23:29,779 INFO [INIT] Marking all workers as offline 2019-10-31 06:23:29,822 INFO [INIT] LAVA master has started. 2019-10-31 06:23:29,822 INFO [INIT] Using protocol version 3 2019-10-31 06:23:39,306 DEBUG lava-logs => PING(20) 2019-10-31 06:23:48,626 DEBUG Dhanu => PING(20) 2019-10-31 06:23:48,627 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:23:49,823 INFO scheduling health checks: 2019-10-31 06:23:49,843 INFO scheduling jobs: 2019-10-31 06:23:49,844 DEBUG - qemu 2019-10-31 06:23:49,912 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:23:49,913 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:23:49,917 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:23:51,036 INFO [INIT] Dropping privileges 2019-10-31 06:23:51,036 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:23:51,036 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:23:51,038 INFO [INIT] Marking all workers as offline 2019-10-31 06:23:51,068 INFO [INIT] LAVA master has started. 2019-10-31 06:23:51,068 INFO [INIT] Using protocol version 3 2019-10-31 06:23:59,328 DEBUG lava-logs => PING(20) 2019-10-31 06:24:08,647 DEBUG Dhanu => PING(20) 2019-10-31 06:24:08,648 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:24:11,071 INFO scheduling health checks: 2019-10-31 06:24:11,123 INFO scheduling jobs: 2019-10-31 06:24:11,126 DEBUG - qemu 2019-10-31 06:24:11,202 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:24:11,202 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:24:11,207 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:24:12,295 INFO [INIT] Dropping privileges 2019-10-31 06:24:12,295 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:24:12,295 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:24:12,296 INFO [INIT] Marking all workers as offline 2019-10-31 06:24:12,326 INFO [INIT] LAVA master has started. 2019-10-31 06:24:12,326 INFO [INIT] Using protocol version 3 2019-10-31 06:24:19,348 DEBUG lava-logs => PING(20) 2019-10-31 06:24:28,664 DEBUG Dhanu => PING(20) 2019-10-31 06:24:28,665 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:24:32,330 INFO scheduling health checks: 2019-10-31 06:24:32,381 INFO scheduling jobs: 2019-10-31 06:24:32,383 DEBUG - qemu 2019-10-31 06:24:32,457 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:24:32,457 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:24:32,461 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:24:33,548 INFO [INIT] Dropping privileges 2019-10-31 06:24:33,548 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:24:33,548 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:24:33,550 INFO [INIT] Marking all workers as offline 2019-10-31 06:24:33,582 INFO [INIT] LAVA master has started. 2019-10-31 06:24:33,582 INFO [INIT] Using protocol version 3 2019-10-31 06:24:39,368 DEBUG lava-logs => PING(20) 2019-10-31 06:24:48,680 DEBUG Dhanu => PING(20) 2019-10-31 06:24:48,681 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:24:51,790 INFO [POLL] Received a signal, leaving 2019-10-31 06:24:51,790 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:24:52,756 INFO [INIT] Dropping privileges 2019-10-31 06:24:52,756 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:24:52,757 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:24:52,758 INFO [INIT] Marking all workers as offline 2019-10-31 06:24:52,785 INFO [INIT] LAVA master has started. 2019-10-31 06:24:52,785 INFO [INIT] Using protocol version 3 2019-10-31 06:24:57,633 INFO Dhanu => HELLO 2019-10-31 06:24:57,633 WARNING New dispatcher 2019-10-31 06:24:59,388 DEBUG lava-logs => PING(20) 2019-10-31 06:25:02,640 DEBUG Dhanu => PING(20) 2019-10-31 06:25:12,795 INFO scheduling health checks: 2019-10-31 06:25:12,929 INFO scheduling jobs: 2019-10-31 06:25:12,932 DEBUG - qemu 2019-10-31 06:25:13,037 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 06:25:13,037 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 06:25:13,043 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 06:25:14,464 INFO [INIT] Dropping privileges 2019-10-31 06:25:14,464 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 06:25:14,464 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 06:25:14,467 INFO [INIT] Marking all workers as offline 2019-10-31 06:25:14,543 INFO [INIT] LAVA master has started. 2019-10-31 06:25:14,543 INFO [INIT] Using protocol version 3 2019-10-31 06:25:14,828 DEBUG lava-logs => PING(20) 2019-10-31 06:25:14,879 DEBUG lava-logs => PING(20) 2019-10-31 06:25:22,656 DEBUG Dhanu => PING(20) 2019-10-31 06:25:22,657 WARNING Unknown dispatcher (server crashed) 2019-10-31 06:25:33,386 INFO [POLL] Received a signal, leaving 2019-10-31 06:25:33,386 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 11:59:41,532 INFO [INIT] Dropping privileges 2019-10-31 11:59:41,608 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 11:59:41,609 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 11:59:41,623 INFO [INIT] Marking all workers as offline 2019-10-31 11:59:41,692 INFO [INIT] LAVA master has started. 2019-10-31 11:59:41,692 INFO [INIT] Using protocol version 3 2019-10-31 11:59:41,717 INFO Dhanu => HELLO 2019-10-31 11:59:41,718 WARNING New dispatcher 2019-10-31 11:59:41,801 INFO Dhanu => HELLO_RETRY 2019-10-31 11:59:41,801 WARNING Dispatcher was not confirmed 2019-10-31 11:59:41,834 INFO Dhanu => HELLO_RETRY 2019-10-31 11:59:41,835 WARNING Dispatcher was not confirmed 2019-10-31 11:59:41,867 DEBUG lava-logs => PING(20) 2019-10-31 11:59:41,901 INFO Dhanu => HELLO_RETRY 2019-10-31 11:59:41,901 WARNING Dispatcher was not confirmed 2019-10-31 11:59:41,934 DEBUG lava-logs => PING(20) 2019-10-31 11:59:41,967 INFO Dhanu => HELLO_RETRY 2019-10-31 11:59:41,968 WARNING Dispatcher was not confirmed 2019-10-31 11:59:46,974 DEBUG Dhanu => PING(20) 2019-10-31 12:00:01,705 INFO scheduling health checks: 2019-10-31 12:00:02,047 INFO scheduling jobs: 2019-10-31 12:00:02,050 DEBUG - qemu 2019-10-31 12:00:02,443 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:00:02,443 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:00:02,505 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:00:03,889 INFO [INIT] Dropping privileges 2019-10-31 12:00:03,889 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:00:03,889 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:00:03,891 INFO [INIT] Marking all workers as offline 2019-10-31 12:00:03,950 INFO [INIT] LAVA master has started. 2019-10-31 12:00:03,951 INFO [INIT] Using protocol version 3 2019-10-31 12:00:06,995 DEBUG Dhanu => PING(20) 2019-10-31 12:00:06,995 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:00:21,976 DEBUG lava-logs => PING(20) 2019-10-31 12:00:23,953 INFO scheduling health checks: 2019-10-31 12:00:24,005 INFO scheduling jobs: 2019-10-31 12:00:24,007 DEBUG - qemu 2019-10-31 12:00:24,081 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:00:24,081 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:00:24,085 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:00:25,364 INFO [INIT] Dropping privileges 2019-10-31 12:00:25,364 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:00:25,364 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:00:25,366 INFO [INIT] Marking all workers as offline 2019-10-31 12:00:25,430 INFO [INIT] LAVA master has started. 2019-10-31 12:00:25,430 INFO [INIT] Using protocol version 3 2019-10-31 12:00:27,015 DEBUG Dhanu => PING(20) 2019-10-31 12:00:27,015 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:00:41,992 DEBUG lava-logs => PING(20) 2019-10-31 12:00:45,435 INFO scheduling health checks: 2019-10-31 12:00:45,496 INFO scheduling jobs: 2019-10-31 12:00:45,499 DEBUG - qemu 2019-10-31 12:00:45,566 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:00:45,566 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:00:45,570 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:00:46,901 INFO [INIT] Dropping privileges 2019-10-31 12:00:46,901 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:00:46,901 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:00:46,903 INFO [INIT] Marking all workers as offline 2019-10-31 12:00:46,964 INFO [INIT] LAVA master has started. 2019-10-31 12:00:46,965 INFO [INIT] Using protocol version 3 2019-10-31 12:00:47,030 DEBUG Dhanu => PING(20) 2019-10-31 12:00:47,031 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:01:02,010 DEBUG lava-logs => PING(20) 2019-10-31 12:01:06,970 INFO scheduling health checks: 2019-10-31 12:01:07,009 INFO scheduling jobs: 2019-10-31 12:01:07,010 DEBUG - qemu 2019-10-31 12:01:07,083 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:01:07,083 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:01:07,087 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:01:08,278 INFO [INIT] Dropping privileges 2019-10-31 12:01:08,278 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:01:08,278 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:01:08,279 INFO [INIT] Marking all workers as offline 2019-10-31 12:01:08,345 INFO [INIT] LAVA master has started. 2019-10-31 12:01:08,345 INFO [INIT] Using protocol version 3 2019-10-31 12:01:22,026 DEBUG lava-logs => PING(20) 2019-10-31 12:01:27,069 DEBUG Dhanu => PING(20) 2019-10-31 12:01:27,070 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:01:28,347 INFO scheduling health checks: 2019-10-31 12:01:28,389 INFO scheduling jobs: 2019-10-31 12:01:28,392 DEBUG - qemu 2019-10-31 12:01:28,461 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:01:28,461 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:01:28,465 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:01:29,729 INFO [INIT] Dropping privileges 2019-10-31 12:01:29,729 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:01:29,729 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:01:29,731 INFO [INIT] Marking all workers as offline 2019-10-31 12:01:29,768 INFO [INIT] LAVA master has started. 2019-10-31 12:01:29,768 INFO [INIT] Using protocol version 3 2019-10-31 12:01:42,038 DEBUG lava-logs => PING(20) 2019-10-31 12:01:47,086 DEBUG Dhanu => PING(20) 2019-10-31 12:01:47,086 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:01:49,769 INFO scheduling health checks: 2019-10-31 12:01:49,869 INFO scheduling jobs: 2019-10-31 12:01:49,871 DEBUG - qemu 2019-10-31 12:01:49,953 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:01:49,953 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:01:49,958 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:01:51,240 INFO [INIT] Dropping privileges 2019-10-31 12:01:51,240 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:01:51,240 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:01:51,242 INFO [INIT] Marking all workers as offline 2019-10-31 12:01:51,304 INFO [INIT] LAVA master has started. 2019-10-31 12:01:51,304 INFO [INIT] Using protocol version 3 2019-10-31 12:02:02,050 DEBUG lava-logs => PING(20) 2019-10-31 12:02:07,102 DEBUG Dhanu => PING(20) 2019-10-31 12:02:07,102 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:02:11,305 INFO scheduling health checks: 2019-10-31 12:02:11,360 INFO scheduling jobs: 2019-10-31 12:02:11,364 DEBUG - qemu 2019-10-31 12:02:11,440 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:02:11,440 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:02:11,445 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:02:12,839 INFO [INIT] Dropping privileges 2019-10-31 12:02:12,839 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:02:12,840 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:02:12,841 INFO [INIT] Marking all workers as offline 2019-10-31 12:02:12,896 INFO [INIT] LAVA master has started. 2019-10-31 12:02:12,896 INFO [INIT] Using protocol version 3 2019-10-31 12:02:22,067 DEBUG lava-logs => PING(20) 2019-10-31 12:02:27,118 DEBUG Dhanu => PING(20) 2019-10-31 12:02:27,119 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:02:32,901 INFO scheduling health checks: 2019-10-31 12:02:32,962 INFO scheduling jobs: 2019-10-31 12:02:32,965 DEBUG - qemu 2019-10-31 12:02:33,039 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:02:33,039 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:02:33,043 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:02:34,380 INFO [INIT] Dropping privileges 2019-10-31 12:02:34,381 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:02:34,381 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:02:34,382 INFO [INIT] Marking all workers as offline 2019-10-31 12:02:34,432 INFO [INIT] LAVA master has started. 2019-10-31 12:02:34,432 INFO [INIT] Using protocol version 3 2019-10-31 12:02:42,086 DEBUG lava-logs => PING(20) 2019-10-31 12:02:47,138 DEBUG Dhanu => PING(20) 2019-10-31 12:02:47,138 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:02:54,440 INFO scheduling health checks: 2019-10-31 12:02:54,502 INFO scheduling jobs: 2019-10-31 12:02:54,506 DEBUG - qemu 2019-10-31 12:02:54,581 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:02:54,581 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:02:54,586 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:02:55,796 INFO [INIT] Dropping privileges 2019-10-31 12:02:55,797 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:02:55,797 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:02:55,798 INFO [INIT] Marking all workers as offline 2019-10-31 12:02:55,856 INFO [INIT] LAVA master has started. 2019-10-31 12:02:55,856 INFO [INIT] Using protocol version 3 2019-10-31 12:03:02,105 DEBUG lava-logs => PING(20) 2019-10-31 12:03:07,152 DEBUG Dhanu => PING(20) 2019-10-31 12:03:07,152 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:03:15,865 INFO scheduling health checks: 2019-10-31 12:03:15,922 INFO scheduling jobs: 2019-10-31 12:03:15,924 DEBUG - qemu 2019-10-31 12:03:15,994 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:03:15,994 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:03:15,999 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:03:17,148 INFO [INIT] Dropping privileges 2019-10-31 12:03:17,148 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:03:17,148 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:03:17,149 INFO [INIT] Marking all workers as offline 2019-10-31 12:03:17,214 INFO [INIT] LAVA master has started. 2019-10-31 12:03:17,214 INFO [INIT] Using protocol version 3 2019-10-31 12:03:22,121 DEBUG lava-logs => PING(20) 2019-10-31 12:03:27,170 DEBUG Dhanu => PING(20) 2019-10-31 12:03:27,171 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:03:37,221 INFO scheduling health checks: 2019-10-31 12:03:37,280 INFO scheduling jobs: 2019-10-31 12:03:37,282 DEBUG - qemu 2019-10-31 12:03:37,353 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:03:37,353 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:03:37,358 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:03:38,629 INFO [INIT] Dropping privileges 2019-10-31 12:03:38,630 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:03:38,630 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:03:38,631 INFO [INIT] Marking all workers as offline 2019-10-31 12:03:38,693 INFO [INIT] LAVA master has started. 2019-10-31 12:03:38,694 INFO [INIT] Using protocol version 3 2019-10-31 12:03:42,134 DEBUG lava-logs => PING(20) 2019-10-31 12:03:47,192 DEBUG Dhanu => PING(20) 2019-10-31 12:03:47,192 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:03:58,705 INFO scheduling health checks: 2019-10-31 12:03:58,760 INFO scheduling jobs: 2019-10-31 12:03:58,763 DEBUG - qemu 2019-10-31 12:03:58,838 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:03:58,838 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:03:58,854 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:04:00,098 INFO [INIT] Dropping privileges 2019-10-31 12:04:00,098 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:04:00,098 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:04:00,100 INFO [INIT] Marking all workers as offline 2019-10-31 12:04:00,162 INFO [INIT] LAVA master has started. 2019-10-31 12:04:00,163 INFO [INIT] Using protocol version 3 2019-10-31 12:04:02,149 DEBUG lava-logs => PING(20) 2019-10-31 12:04:07,211 DEBUG Dhanu => PING(20) 2019-10-31 12:04:07,211 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:04:20,173 INFO scheduling health checks: 2019-10-31 12:04:20,206 INFO scheduling jobs: 2019-10-31 12:04:20,209 DEBUG - qemu 2019-10-31 12:04:20,280 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:04:20,280 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:04:20,285 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:04:21,340 INFO [INIT] Dropping privileges 2019-10-31 12:04:21,340 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:04:21,341 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:04:21,342 INFO [INIT] Marking all workers as offline 2019-10-31 12:04:21,375 INFO [INIT] LAVA master has started. 2019-10-31 12:04:21,376 INFO [INIT] Using protocol version 3 2019-10-31 12:04:22,162 DEBUG lava-logs => PING(20) 2019-10-31 12:04:27,231 DEBUG Dhanu => PING(20) 2019-10-31 12:04:27,231 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:04:28,617 INFO [POLL] Received a signal, leaving 2019-10-31 12:04:28,617 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:04:29,589 INFO [INIT] Dropping privileges 2019-10-31 12:04:29,589 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:04:29,589 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:04:29,591 INFO [INIT] Marking all workers as offline 2019-10-31 12:04:29,620 INFO [INIT] LAVA master has started. 2019-10-31 12:04:29,621 INFO [INIT] Using protocol version 3 2019-10-31 12:04:47,245 DEBUG Dhanu => PING(20) 2019-10-31 12:04:47,246 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:04:47,286 DEBUG lava-logs => PING(20) 2019-10-31 12:04:47,298 DEBUG lava-logs => PING(20) 2019-10-31 12:04:49,623 INFO scheduling health checks: 2019-10-31 12:04:49,653 INFO scheduling jobs: 2019-10-31 12:04:49,655 DEBUG - qemu 2019-10-31 12:04:49,728 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:04:49,728 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:04:49,732 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:04:50,842 INFO [INIT] Dropping privileges 2019-10-31 12:04:50,842 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:04:50,842 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:04:50,844 INFO [INIT] Marking all workers as offline 2019-10-31 12:04:50,878 INFO [INIT] LAVA master has started. 2019-10-31 12:04:50,879 INFO [INIT] Using protocol version 3 2019-10-31 12:05:07,264 DEBUG Dhanu => PING(20) 2019-10-31 12:05:07,265 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:05:07,322 DEBUG lava-logs => PING(20) 2019-10-31 12:05:10,883 INFO scheduling health checks: 2019-10-31 12:05:10,934 INFO scheduling jobs: 2019-10-31 12:05:10,937 DEBUG - qemu 2019-10-31 12:05:11,014 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:05:11,014 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:05:11,019 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:05:12,108 INFO [INIT] Dropping privileges 2019-10-31 12:05:12,109 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:05:12,109 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:05:12,111 INFO [INIT] Marking all workers as offline 2019-10-31 12:05:12,147 INFO [INIT] LAVA master has started. 2019-10-31 12:05:12,147 INFO [INIT] Using protocol version 3 2019-10-31 12:05:23,701 INFO Dhanu => HELLO 2019-10-31 12:05:23,702 WARNING New dispatcher 2019-10-31 12:05:27,338 DEBUG lava-logs => PING(20) 2019-10-31 12:05:28,708 DEBUG Dhanu => PING(20) 2019-10-31 12:05:32,149 INFO scheduling health checks: 2019-10-31 12:05:32,180 INFO scheduling jobs: 2019-10-31 12:05:32,182 DEBUG - qemu 2019-10-31 12:05:32,255 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:05:32,255 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:05:32,260 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:05:33,356 INFO [INIT] Dropping privileges 2019-10-31 12:05:33,356 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:05:33,356 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:05:33,358 INFO [INIT] Marking all workers as offline 2019-10-31 12:05:33,416 INFO [INIT] LAVA master has started. 2019-10-31 12:05:33,417 INFO [INIT] Using protocol version 3 2019-10-31 12:05:47,354 DEBUG lava-logs => PING(20) 2019-10-31 12:05:48,726 DEBUG Dhanu => PING(20) 2019-10-31 12:05:48,727 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:05:53,418 INFO scheduling health checks: 2019-10-31 12:05:53,437 INFO scheduling jobs: 2019-10-31 12:05:53,439 DEBUG - qemu 2019-10-31 12:05:53,508 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:05:53,508 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:05:53,512 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:05:54,773 INFO [INIT] Dropping privileges 2019-10-31 12:05:54,774 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:05:54,774 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:05:54,775 INFO [INIT] Marking all workers as offline 2019-10-31 12:05:54,805 INFO [INIT] LAVA master has started. 2019-10-31 12:05:54,806 INFO [INIT] Using protocol version 3 2019-10-31 12:06:07,366 DEBUG lava-logs => PING(20) 2019-10-31 12:06:08,742 DEBUG Dhanu => PING(20) 2019-10-31 12:06:08,743 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:06:14,812 INFO scheduling health checks: 2019-10-31 12:06:14,896 INFO scheduling jobs: 2019-10-31 12:06:14,899 DEBUG - qemu 2019-10-31 12:06:14,975 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:06:14,976 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:06:14,980 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:06:16,097 INFO [INIT] Dropping privileges 2019-10-31 12:06:16,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:06:16,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:06:16,098 INFO [INIT] Marking all workers as offline 2019-10-31 12:06:16,131 INFO [INIT] LAVA master has started. 2019-10-31 12:06:16,131 INFO [INIT] Using protocol version 3 2019-10-31 12:06:27,377 DEBUG lava-logs => PING(20) 2019-10-31 12:06:28,763 DEBUG Dhanu => PING(20) 2019-10-31 12:06:28,763 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:06:36,137 INFO scheduling health checks: 2019-10-31 12:06:36,164 INFO scheduling jobs: 2019-10-31 12:06:36,166 DEBUG - qemu 2019-10-31 12:06:36,258 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:06:36,259 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:06:36,263 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:06:37,534 INFO [INIT] Dropping privileges 2019-10-31 12:06:37,534 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:06:37,534 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:06:37,536 INFO [INIT] Marking all workers as offline 2019-10-31 12:06:37,566 INFO [INIT] LAVA master has started. 2019-10-31 12:06:37,566 INFO [INIT] Using protocol version 3 2019-10-31 12:06:47,394 DEBUG lava-logs => PING(20) 2019-10-31 12:06:48,779 DEBUG Dhanu => PING(20) 2019-10-31 12:06:48,780 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:06:57,575 INFO scheduling health checks: 2019-10-31 12:06:57,633 INFO scheduling jobs: 2019-10-31 12:06:57,634 DEBUG - qemu 2019-10-31 12:06:57,700 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:06:57,701 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:06:57,705 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:06:58,846 INFO [INIT] Dropping privileges 2019-10-31 12:06:58,846 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:06:58,846 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:06:58,848 INFO [INIT] Marking all workers as offline 2019-10-31 12:06:58,879 INFO [INIT] LAVA master has started. 2019-10-31 12:06:58,880 INFO [INIT] Using protocol version 3 2019-10-31 12:07:07,414 DEBUG lava-logs => PING(20) 2019-10-31 12:07:08,803 DEBUG Dhanu => PING(20) 2019-10-31 12:07:08,804 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:07:18,889 INFO scheduling health checks: 2019-10-31 12:07:18,935 INFO scheduling jobs: 2019-10-31 12:07:18,938 DEBUG - qemu 2019-10-31 12:07:19,013 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:07:19,013 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:07:19,017 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:07:20,107 INFO [INIT] Dropping privileges 2019-10-31 12:07:20,108 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:07:20,108 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:07:20,109 INFO [INIT] Marking all workers as offline 2019-10-31 12:07:20,171 INFO [INIT] LAVA master has started. 2019-10-31 12:07:20,172 INFO [INIT] Using protocol version 3 2019-10-31 12:07:27,430 DEBUG lava-logs => PING(20) 2019-10-31 12:07:28,826 DEBUG Dhanu => PING(20) 2019-10-31 12:07:28,827 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:07:40,183 INFO scheduling health checks: 2019-10-31 12:07:40,226 INFO scheduling jobs: 2019-10-31 12:07:40,228 DEBUG - qemu 2019-10-31 12:07:40,303 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:07:40,303 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:07:40,308 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:07:41,583 INFO [INIT] Dropping privileges 2019-10-31 12:07:41,584 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:07:41,584 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:07:41,585 INFO [INIT] Marking all workers as offline 2019-10-31 12:07:41,617 INFO [INIT] LAVA master has started. 2019-10-31 12:07:41,618 INFO [INIT] Using protocol version 3 2019-10-31 12:07:47,446 DEBUG lava-logs => PING(20) 2019-10-31 12:07:48,847 DEBUG Dhanu => PING(20) 2019-10-31 12:07:48,848 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:08:01,631 INFO scheduling health checks: 2019-10-31 12:08:01,673 INFO scheduling jobs: 2019-10-31 12:08:01,676 DEBUG - qemu 2019-10-31 12:08:01,750 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:08:01,750 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:08:01,755 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:08:02,865 INFO [INIT] Dropping privileges 2019-10-31 12:08:02,865 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:08:02,865 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:08:02,867 INFO [INIT] Marking all workers as offline 2019-10-31 12:08:02,898 INFO [INIT] LAVA master has started. 2019-10-31 12:08:02,898 INFO [INIT] Using protocol version 3 2019-10-31 12:08:07,462 DEBUG lava-logs => PING(20) 2019-10-31 12:08:08,867 DEBUG Dhanu => PING(20) 2019-10-31 12:08:08,867 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:08:22,912 INFO scheduling health checks: 2019-10-31 12:08:22,952 INFO scheduling jobs: 2019-10-31 12:08:22,955 DEBUG - qemu 2019-10-31 12:08:23,029 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:08:23,029 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:08:23,033 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:08:24,127 INFO [INIT] Dropping privileges 2019-10-31 12:08:24,128 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:08:24,128 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:08:24,129 INFO [INIT] Marking all workers as offline 2019-10-31 12:08:24,189 INFO [INIT] LAVA master has started. 2019-10-31 12:08:24,189 INFO [INIT] Using protocol version 3 2019-10-31 12:08:27,478 DEBUG lava-logs => PING(20) 2019-10-31 12:08:28,890 DEBUG Dhanu => PING(20) 2019-10-31 12:08:28,891 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:08:44,205 INFO scheduling health checks: 2019-10-31 12:08:44,243 INFO scheduling jobs: 2019-10-31 12:08:44,246 DEBUG - qemu 2019-10-31 12:08:44,319 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:08:44,319 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:08:44,326 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:08:45,601 INFO [INIT] Dropping privileges 2019-10-31 12:08:45,601 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:08:45,601 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:08:45,603 INFO [INIT] Marking all workers as offline 2019-10-31 12:08:45,634 INFO [INIT] LAVA master has started. 2019-10-31 12:08:45,634 INFO [INIT] Using protocol version 3 2019-10-31 12:08:47,494 DEBUG lava-logs => PING(20) 2019-10-31 12:08:48,910 DEBUG Dhanu => PING(20) 2019-10-31 12:08:48,911 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:09:05,651 INFO scheduling health checks: 2019-10-31 12:09:05,690 INFO scheduling jobs: 2019-10-31 12:09:05,694 DEBUG - qemu 2019-10-31 12:09:05,768 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:09:05,768 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:09:05,778 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:09:06,837 INFO [INIT] Dropping privileges 2019-10-31 12:09:06,837 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:09:06,837 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:09:06,838 INFO [INIT] Marking all workers as offline 2019-10-31 12:09:06,869 INFO [INIT] LAVA master has started. 2019-10-31 12:09:06,869 INFO [INIT] Using protocol version 3 2019-10-31 12:09:07,507 DEBUG lava-logs => PING(20) 2019-10-31 12:09:08,930 DEBUG Dhanu => PING(20) 2019-10-31 12:09:08,931 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:09:26,885 INFO scheduling health checks: 2019-10-31 12:09:26,925 INFO scheduling jobs: 2019-10-31 12:09:26,928 DEBUG - qemu 2019-10-31 12:09:27,002 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:09:27,002 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:09:27,006 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:09:28,095 INFO [INIT] Dropping privileges 2019-10-31 12:09:28,095 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:09:28,095 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:09:28,096 INFO [INIT] Marking all workers as offline 2019-10-31 12:09:28,127 INFO [INIT] LAVA master has started. 2019-10-31 12:09:28,127 INFO [INIT] Using protocol version 3 2019-10-31 12:09:28,176 DEBUG lava-logs => PING(20) 2019-10-31 12:09:28,951 DEBUG Dhanu => PING(20) 2019-10-31 12:09:28,951 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:09:48,147 INFO scheduling health checks: 2019-10-31 12:09:48,183 INFO scheduling jobs: 2019-10-31 12:09:48,186 DEBUG - qemu 2019-10-31 12:09:48,254 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:09:48,254 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:09:48,264 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:09:49,341 INFO [INIT] Dropping privileges 2019-10-31 12:09:49,342 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:09:49,342 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:09:49,343 INFO [INIT] Marking all workers as offline 2019-10-31 12:09:49,374 INFO [INIT] LAVA master has started. 2019-10-31 12:09:49,374 INFO [INIT] Using protocol version 3 2019-10-31 12:09:49,494 DEBUG Dhanu => PING(20) 2019-10-31 12:09:49,495 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:10:08,219 DEBUG lava-logs => PING(20) 2019-10-31 12:10:09,376 INFO scheduling health checks: 2019-10-31 12:10:09,407 INFO scheduling jobs: 2019-10-31 12:10:09,409 DEBUG - qemu 2019-10-31 12:10:09,483 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:10:09,484 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:10:09,488 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:10:10,597 INFO [INIT] Dropping privileges 2019-10-31 12:10:10,597 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:10:10,597 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:10:10,599 INFO [INIT] Marking all workers as offline 2019-10-31 12:10:10,666 INFO [INIT] LAVA master has started. 2019-10-31 12:10:10,666 INFO [INIT] Using protocol version 3 2019-10-31 12:10:10,778 DEBUG Dhanu => PING(20) 2019-10-31 12:10:10,779 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:10:28,230 DEBUG lava-logs => PING(20) 2019-10-31 12:10:30,668 INFO scheduling health checks: 2019-10-31 12:10:30,698 INFO scheduling jobs: 2019-10-31 12:10:30,700 DEBUG - qemu 2019-10-31 12:10:30,766 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:10:30,766 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:10:30,776 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:10:31,849 INFO [INIT] Dropping privileges 2019-10-31 12:10:31,849 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:10:31,849 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:10:31,851 INFO [INIT] Marking all workers as offline 2019-10-31 12:10:31,890 INFO [INIT] LAVA master has started. 2019-10-31 12:10:31,891 INFO [INIT] Using protocol version 3 2019-10-31 12:10:31,951 DEBUG Dhanu => PING(20) 2019-10-31 12:10:31,951 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:10:48,246 DEBUG lava-logs => PING(20) 2019-10-31 12:10:51,895 INFO scheduling health checks: 2019-10-31 12:10:51,923 INFO scheduling jobs: 2019-10-31 12:10:51,925 DEBUG - qemu 2019-10-31 12:10:51,999 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:10:51,999 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:10:52,008 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:10:53,089 INFO [INIT] Dropping privileges 2019-10-31 12:10:53,090 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:10:53,090 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:10:53,091 INFO [INIT] Marking all workers as offline 2019-10-31 12:10:53,125 INFO [INIT] LAVA master has started. 2019-10-31 12:10:53,126 INFO [INIT] Using protocol version 3 2019-10-31 12:11:08,265 DEBUG lava-logs => PING(20) 2019-10-31 12:11:11,991 DEBUG Dhanu => PING(20) 2019-10-31 12:11:11,991 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:11:13,128 INFO scheduling health checks: 2019-10-31 12:11:13,147 INFO scheduling jobs: 2019-10-31 12:11:13,148 DEBUG - qemu 2019-10-31 12:11:13,218 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:11:13,218 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:11:13,231 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:11:14,340 INFO [INIT] Dropping privileges 2019-10-31 12:11:14,341 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:11:14,341 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:11:14,342 INFO [INIT] Marking all workers as offline 2019-10-31 12:11:14,372 INFO [INIT] LAVA master has started. 2019-10-31 12:11:14,372 INFO [INIT] Using protocol version 3 2019-10-31 12:11:28,281 DEBUG lava-logs => PING(20) 2019-10-31 12:11:32,013 DEBUG Dhanu => PING(20) 2019-10-31 12:11:32,013 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:11:34,374 INFO scheduling health checks: 2019-10-31 12:11:34,416 INFO scheduling jobs: 2019-10-31 12:11:34,419 DEBUG - qemu 2019-10-31 12:11:34,489 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:11:34,489 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:11:34,493 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:11:35,596 INFO [INIT] Dropping privileges 2019-10-31 12:11:35,596 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:11:35,597 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:11:35,598 INFO [INIT] Marking all workers as offline 2019-10-31 12:11:35,629 INFO [INIT] LAVA master has started. 2019-10-31 12:11:35,629 INFO [INIT] Using protocol version 3 2019-10-31 12:11:48,297 DEBUG lava-logs => PING(20) 2019-10-31 12:11:52,034 DEBUG Dhanu => PING(20) 2019-10-31 12:11:52,034 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:11:55,633 INFO scheduling health checks: 2019-10-31 12:11:55,662 INFO scheduling jobs: 2019-10-31 12:11:55,664 DEBUG - qemu 2019-10-31 12:11:55,736 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:11:55,736 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:11:55,741 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:11:56,841 INFO [INIT] Dropping privileges 2019-10-31 12:11:56,841 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:11:56,841 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:11:56,843 INFO [INIT] Marking all workers as offline 2019-10-31 12:11:56,875 INFO [INIT] LAVA master has started. 2019-10-31 12:11:56,876 INFO [INIT] Using protocol version 3 2019-10-31 12:12:08,312 DEBUG lava-logs => PING(20) 2019-10-31 12:12:12,050 DEBUG Dhanu => PING(20) 2019-10-31 12:12:12,050 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:12:16,877 INFO scheduling health checks: 2019-10-31 12:12:16,898 INFO scheduling jobs: 2019-10-31 12:12:16,899 DEBUG - qemu 2019-10-31 12:12:16,971 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:12:16,971 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:12:16,975 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:12:18,102 INFO [INIT] Dropping privileges 2019-10-31 12:12:18,102 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:12:18,102 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:12:18,104 INFO [INIT] Marking all workers as offline 2019-10-31 12:12:18,133 INFO [INIT] LAVA master has started. 2019-10-31 12:12:18,133 INFO [INIT] Using protocol version 3 2019-10-31 12:12:28,322 DEBUG lava-logs => PING(20) 2019-10-31 12:12:32,068 DEBUG Dhanu => PING(20) 2019-10-31 12:12:32,068 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:12:38,139 INFO scheduling health checks: 2019-10-31 12:12:38,178 INFO scheduling jobs: 2019-10-31 12:12:38,181 DEBUG - qemu 2019-10-31 12:12:38,255 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:12:38,255 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:12:38,259 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:12:39,344 INFO [INIT] Dropping privileges 2019-10-31 12:12:39,344 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:12:39,344 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:12:39,346 INFO [INIT] Marking all workers as offline 2019-10-31 12:12:39,380 INFO [INIT] LAVA master has started. 2019-10-31 12:12:39,381 INFO [INIT] Using protocol version 3 2019-10-31 12:12:48,343 DEBUG lava-logs => PING(20) 2019-10-31 12:12:52,089 DEBUG Dhanu => PING(20) 2019-10-31 12:12:52,089 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:12:59,388 INFO scheduling health checks: 2019-10-31 12:12:59,413 INFO scheduling jobs: 2019-10-31 12:12:59,415 DEBUG - qemu 2019-10-31 12:12:59,487 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:12:59,487 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:12:59,491 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:13:00,592 INFO [INIT] Dropping privileges 2019-10-31 12:13:00,592 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:13:00,592 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:13:00,594 INFO [INIT] Marking all workers as offline 2019-10-31 12:13:00,627 INFO [INIT] LAVA master has started. 2019-10-31 12:13:00,628 INFO [INIT] Using protocol version 3 2019-10-31 12:13:08,362 DEBUG lava-logs => PING(20) 2019-10-31 12:13:12,109 DEBUG Dhanu => PING(20) 2019-10-31 12:13:12,109 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:13:20,637 INFO scheduling health checks: 2019-10-31 12:13:20,660 INFO scheduling jobs: 2019-10-31 12:13:20,661 DEBUG - qemu 2019-10-31 12:13:20,732 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:13:20,732 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:13:20,737 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:13:21,851 INFO [INIT] Dropping privileges 2019-10-31 12:13:21,851 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:13:21,851 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:13:21,853 INFO [INIT] Marking all workers as offline 2019-10-31 12:13:21,884 INFO [INIT] LAVA master has started. 2019-10-31 12:13:21,884 INFO [INIT] Using protocol version 3 2019-10-31 12:13:28,381 DEBUG lava-logs => PING(20) 2019-10-31 12:13:32,128 DEBUG Dhanu => PING(20) 2019-10-31 12:13:32,129 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:13:41,894 INFO scheduling health checks: 2019-10-31 12:13:41,917 INFO scheduling jobs: 2019-10-31 12:13:41,919 DEBUG - qemu 2019-10-31 12:13:41,989 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:13:41,989 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:13:41,994 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:13:43,107 INFO [INIT] Dropping privileges 2019-10-31 12:13:43,107 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:13:43,107 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:13:43,109 INFO [INIT] Marking all workers as offline 2019-10-31 12:13:43,142 INFO [INIT] LAVA master has started. 2019-10-31 12:13:43,142 INFO [INIT] Using protocol version 3 2019-10-31 12:13:48,399 DEBUG lava-logs => PING(20) 2019-10-31 12:13:52,148 DEBUG Dhanu => PING(20) 2019-10-31 12:13:52,149 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:14:03,153 INFO scheduling health checks: 2019-10-31 12:14:03,175 INFO scheduling jobs: 2019-10-31 12:14:03,176 DEBUG - qemu 2019-10-31 12:14:03,248 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:14:03,248 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:14:03,253 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:14:04,378 INFO [INIT] Dropping privileges 2019-10-31 12:14:04,378 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:14:04,378 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:14:04,380 INFO [INIT] Marking all workers as offline 2019-10-31 12:14:04,455 INFO [INIT] LAVA master has started. 2019-10-31 12:14:04,456 INFO [INIT] Using protocol version 3 2019-10-31 12:14:08,410 DEBUG lava-logs => PING(20) 2019-10-31 12:14:12,170 DEBUG Dhanu => PING(20) 2019-10-31 12:14:12,171 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:14:24,468 INFO scheduling health checks: 2019-10-31 12:14:24,488 INFO scheduling jobs: 2019-10-31 12:14:24,489 DEBUG - qemu 2019-10-31 12:14:24,559 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:14:24,559 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:14:24,564 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:14:25,743 INFO [INIT] Dropping privileges 2019-10-31 12:14:25,743 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:14:25,744 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:14:25,745 INFO [INIT] Marking all workers as offline 2019-10-31 12:14:25,824 INFO [INIT] LAVA master has started. 2019-10-31 12:14:25,825 INFO [INIT] Using protocol version 3 2019-10-31 12:14:28,426 DEBUG lava-logs => PING(20) 2019-10-31 12:14:32,192 DEBUG Dhanu => PING(20) 2019-10-31 12:14:32,192 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:14:45,839 INFO scheduling health checks: 2019-10-31 12:14:45,879 INFO scheduling jobs: 2019-10-31 12:14:45,883 DEBUG - qemu 2019-10-31 12:14:45,955 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:14:45,955 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:14:45,959 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:14:47,089 INFO [INIT] Dropping privileges 2019-10-31 12:14:47,089 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:14:47,089 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:14:47,091 INFO [INIT] Marking all workers as offline 2019-10-31 12:14:47,126 INFO [INIT] LAVA master has started. 2019-10-31 12:14:47,127 INFO [INIT] Using protocol version 3 2019-10-31 12:14:48,440 DEBUG lava-logs => PING(20) 2019-10-31 12:14:52,212 DEBUG Dhanu => PING(20) 2019-10-31 12:14:52,212 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:15:07,142 INFO scheduling health checks: 2019-10-31 12:15:07,195 INFO scheduling jobs: 2019-10-31 12:15:07,198 DEBUG - qemu 2019-10-31 12:15:07,273 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:15:07,273 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:15:07,277 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:15:08,355 INFO [INIT] Dropping privileges 2019-10-31 12:15:08,355 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:15:08,355 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:15:08,357 INFO [INIT] Marking all workers as offline 2019-10-31 12:15:08,430 INFO [INIT] LAVA master has started. 2019-10-31 12:15:08,431 INFO [INIT] Using protocol version 3 2019-10-31 12:15:08,617 DEBUG lava-logs => PING(20) 2019-10-31 12:15:12,234 DEBUG Dhanu => PING(20) 2019-10-31 12:15:12,235 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:15:28,447 INFO scheduling health checks: 2019-10-31 12:15:28,486 INFO scheduling jobs: 2019-10-31 12:15:28,489 DEBUG - qemu 2019-10-31 12:15:28,563 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:15:28,563 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:15:28,568 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:15:29,845 INFO [INIT] Dropping privileges 2019-10-31 12:15:29,845 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:15:29,845 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:15:29,847 INFO [INIT] Marking all workers as offline 2019-10-31 12:15:29,888 INFO [INIT] LAVA master has started. 2019-10-31 12:15:29,888 INFO [INIT] Using protocol version 3 2019-10-31 12:15:29,918 DEBUG lava-logs => PING(20) 2019-10-31 12:15:32,255 DEBUG Dhanu => PING(20) 2019-10-31 12:15:32,256 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:15:49,906 INFO scheduling health checks: 2019-10-31 12:15:49,943 INFO scheduling jobs: 2019-10-31 12:15:49,945 DEBUG - qemu 2019-10-31 12:15:50,014 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:15:50,015 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:15:50,019 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:15:51,090 INFO [INIT] Dropping privileges 2019-10-31 12:15:51,090 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:15:51,090 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:15:51,092 INFO [INIT] Marking all workers as offline 2019-10-31 12:15:51,123 INFO [INIT] LAVA master has started. 2019-10-31 12:15:51,123 INFO [INIT] Using protocol version 3 2019-10-31 12:15:52,275 DEBUG Dhanu => PING(20) 2019-10-31 12:15:52,275 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:16:09,953 DEBUG lava-logs => PING(20) 2019-10-31 12:16:11,125 INFO scheduling health checks: 2019-10-31 12:16:11,145 INFO scheduling jobs: 2019-10-31 12:16:11,146 DEBUG - qemu 2019-10-31 12:16:11,219 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:16:11,219 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:16:11,223 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:16:12,335 INFO [INIT] Dropping privileges 2019-10-31 12:16:12,335 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:16:12,335 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:16:12,337 INFO [INIT] Marking all workers as offline 2019-10-31 12:16:12,369 INFO [INIT] LAVA master has started. 2019-10-31 12:16:12,369 INFO [INIT] Using protocol version 3 2019-10-31 12:16:12,394 DEBUG Dhanu => PING(20) 2019-10-31 12:16:12,394 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:16:29,973 DEBUG lava-logs => PING(20) 2019-10-31 12:16:32,371 INFO scheduling health checks: 2019-10-31 12:16:32,392 INFO scheduling jobs: 2019-10-31 12:16:32,393 DEBUG - qemu 2019-10-31 12:16:32,461 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:16:32,462 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:16:32,466 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:16:33,616 INFO [INIT] Dropping privileges 2019-10-31 12:16:33,617 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:16:33,617 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:16:33,618 INFO [INIT] Marking all workers as offline 2019-10-31 12:16:33,661 INFO [INIT] LAVA master has started. 2019-10-31 12:16:33,662 INFO [INIT] Using protocol version 3 2019-10-31 12:16:49,992 DEBUG lava-logs => PING(20) 2019-10-31 12:16:52,435 DEBUG Dhanu => PING(20) 2019-10-31 12:16:52,436 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:16:53,663 INFO scheduling health checks: 2019-10-31 12:16:53,682 INFO scheduling jobs: 2019-10-31 12:16:53,684 DEBUG - qemu 2019-10-31 12:16:53,748 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:16:53,748 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:16:53,752 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:16:54,825 INFO [INIT] Dropping privileges 2019-10-31 12:16:54,825 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:16:54,825 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:16:54,827 INFO [INIT] Marking all workers as offline 2019-10-31 12:16:54,863 INFO [INIT] LAVA master has started. 2019-10-31 12:16:54,863 INFO [INIT] Using protocol version 3 2019-10-31 12:17:10,010 DEBUG lava-logs => PING(20) 2019-10-31 12:17:12,453 DEBUG Dhanu => PING(20) 2019-10-31 12:17:12,454 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:17:14,865 INFO scheduling health checks: 2019-10-31 12:17:14,884 INFO scheduling jobs: 2019-10-31 12:17:14,886 DEBUG - qemu 2019-10-31 12:17:14,953 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:17:14,954 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:17:14,962 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:17:16,110 INFO [INIT] Dropping privileges 2019-10-31 12:17:16,110 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:17:16,110 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:17:16,112 INFO [INIT] Marking all workers as offline 2019-10-31 12:17:16,152 INFO [INIT] LAVA master has started. 2019-10-31 12:17:16,153 INFO [INIT] Using protocol version 3 2019-10-31 12:17:30,022 DEBUG lava-logs => PING(20) 2019-10-31 12:17:32,470 DEBUG Dhanu => PING(20) 2019-10-31 12:17:32,471 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:17:36,153 INFO scheduling health checks: 2019-10-31 12:17:36,187 INFO scheduling jobs: 2019-10-31 12:17:36,190 DEBUG - qemu 2019-10-31 12:17:36,265 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:17:36,266 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:17:36,270 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:17:37,379 INFO [INIT] Dropping privileges 2019-10-31 12:17:37,380 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:17:37,380 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:17:37,381 INFO [INIT] Marking all workers as offline 2019-10-31 12:17:37,411 INFO [INIT] LAVA master has started. 2019-10-31 12:17:37,411 INFO [INIT] Using protocol version 3 2019-10-31 12:17:50,038 DEBUG lava-logs => PING(20) 2019-10-31 12:17:52,490 DEBUG Dhanu => PING(20) 2019-10-31 12:17:52,490 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:17:57,416 INFO scheduling health checks: 2019-10-31 12:17:57,456 INFO scheduling jobs: 2019-10-31 12:17:57,459 DEBUG - qemu 2019-10-31 12:17:57,532 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:17:57,533 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:17:57,542 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:17:58,847 INFO [INIT] Dropping privileges 2019-10-31 12:17:58,847 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:17:58,847 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:17:58,849 INFO [INIT] Marking all workers as offline 2019-10-31 12:17:58,880 INFO [INIT] LAVA master has started. 2019-10-31 12:17:58,881 INFO [INIT] Using protocol version 3 2019-10-31 12:18:10,051 DEBUG lava-logs => PING(20) 2019-10-31 12:18:12,510 DEBUG Dhanu => PING(20) 2019-10-31 12:18:12,511 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:18:18,888 INFO scheduling health checks: 2019-10-31 12:18:18,924 INFO scheduling jobs: 2019-10-31 12:18:18,927 DEBUG - qemu 2019-10-31 12:18:19,001 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:18:19,001 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:18:19,010 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:18:20,095 INFO [INIT] Dropping privileges 2019-10-31 12:18:20,095 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:18:20,095 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:18:20,097 INFO [INIT] Marking all workers as offline 2019-10-31 12:18:20,126 INFO [INIT] LAVA master has started. 2019-10-31 12:18:20,126 INFO [INIT] Using protocol version 3 2019-10-31 12:18:30,074 DEBUG lava-logs => PING(20) 2019-10-31 12:18:32,532 DEBUG Dhanu => PING(20) 2019-10-31 12:18:32,533 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:18:40,134 INFO scheduling health checks: 2019-10-31 12:18:40,171 INFO scheduling jobs: 2019-10-31 12:18:40,173 DEBUG - qemu 2019-10-31 12:18:40,247 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:18:40,247 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:18:40,252 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:18:41,345 INFO [INIT] Dropping privileges 2019-10-31 12:18:41,345 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:18:41,346 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:18:41,347 INFO [INIT] Marking all workers as offline 2019-10-31 12:18:41,385 INFO [INIT] LAVA master has started. 2019-10-31 12:18:41,385 INFO [INIT] Using protocol version 3 2019-10-31 12:18:50,096 DEBUG lava-logs => PING(20) 2019-10-31 12:18:52,553 DEBUG Dhanu => PING(20) 2019-10-31 12:18:52,554 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:19:01,395 INFO scheduling health checks: 2019-10-31 12:19:01,440 INFO scheduling jobs: 2019-10-31 12:19:01,443 DEBUG - qemu 2019-10-31 12:19:01,517 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:19:01,517 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:19:01,533 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:19:02,622 INFO [INIT] Dropping privileges 2019-10-31 12:19:02,623 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:19:02,623 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:19:02,624 INFO [INIT] Marking all workers as offline 2019-10-31 12:19:02,698 INFO [INIT] LAVA master has started. 2019-10-31 12:19:02,698 INFO [INIT] Using protocol version 3 2019-10-31 12:19:10,116 DEBUG lava-logs => PING(20) 2019-10-31 12:19:12,573 DEBUG Dhanu => PING(20) 2019-10-31 12:19:12,574 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:19:22,709 INFO scheduling health checks: 2019-10-31 12:19:22,742 INFO scheduling jobs: 2019-10-31 12:19:22,744 DEBUG - qemu 2019-10-31 12:19:22,815 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:19:22,816 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:19:22,820 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:19:24,089 INFO [INIT] Dropping privileges 2019-10-31 12:19:24,089 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:19:24,089 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:19:24,091 INFO [INIT] Marking all workers as offline 2019-10-31 12:19:24,133 INFO [INIT] LAVA master has started. 2019-10-31 12:19:24,134 INFO [INIT] Using protocol version 3 2019-10-31 12:19:30,136 DEBUG lava-logs => PING(20) 2019-10-31 12:19:32,596 DEBUG Dhanu => PING(20) 2019-10-31 12:19:32,596 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:19:44,146 INFO scheduling health checks: 2019-10-31 12:19:44,177 INFO scheduling jobs: 2019-10-31 12:19:44,179 DEBUG - qemu 2019-10-31 12:19:44,253 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:19:44,253 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:19:44,258 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:19:45,346 INFO [INIT] Dropping privileges 2019-10-31 12:19:45,346 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:19:45,347 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:19:45,348 INFO [INIT] Marking all workers as offline 2019-10-31 12:19:45,380 INFO [INIT] LAVA master has started. 2019-10-31 12:19:45,380 INFO [INIT] Using protocol version 3 2019-10-31 12:19:50,154 DEBUG lava-logs => PING(20) 2019-10-31 12:19:52,617 DEBUG Dhanu => PING(20) 2019-10-31 12:19:52,618 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:20:05,393 INFO scheduling health checks: 2019-10-31 12:20:05,424 INFO scheduling jobs: 2019-10-31 12:20:05,426 DEBUG - qemu 2019-10-31 12:20:05,499 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:20:05,500 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:20:05,511 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:20:06,592 INFO [INIT] Dropping privileges 2019-10-31 12:20:06,592 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:20:06,592 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:20:06,594 INFO [INIT] Marking all workers as offline 2019-10-31 12:20:06,626 INFO [INIT] LAVA master has started. 2019-10-31 12:20:06,626 INFO [INIT] Using protocol version 3 2019-10-31 12:20:10,166 DEBUG lava-logs => PING(20) 2019-10-31 12:20:12,637 DEBUG Dhanu => PING(20) 2019-10-31 12:20:12,638 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:20:26,641 INFO scheduling health checks: 2019-10-31 12:20:26,681 INFO scheduling jobs: 2019-10-31 12:20:26,683 DEBUG - qemu 2019-10-31 12:20:26,756 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:20:26,757 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:20:26,761 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:20:27,840 INFO [INIT] Dropping privileges 2019-10-31 12:20:27,840 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:20:27,840 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:20:27,842 INFO [INIT] Marking all workers as offline 2019-10-31 12:20:27,884 INFO [INIT] LAVA master has started. 2019-10-31 12:20:27,884 INFO [INIT] Using protocol version 3 2019-10-31 12:20:30,173 DEBUG lava-logs => PING(20) 2019-10-31 12:20:32,660 DEBUG Dhanu => PING(20) 2019-10-31 12:20:32,660 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:20:47,899 INFO scheduling health checks: 2019-10-31 12:20:47,972 INFO scheduling jobs: 2019-10-31 12:20:47,976 DEBUG - qemu 2019-10-31 12:20:48,050 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:20:48,050 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:20:48,059 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:20:49,377 INFO [INIT] Dropping privileges 2019-10-31 12:20:49,377 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:20:49,378 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:20:49,379 INFO [INIT] Marking all workers as offline 2019-10-31 12:20:49,453 INFO [INIT] LAVA master has started. 2019-10-31 12:20:49,453 INFO [INIT] Using protocol version 3 2019-10-31 12:20:50,179 DEBUG lava-logs => PING(20) 2019-10-31 12:20:52,680 DEBUG Dhanu => PING(20) 2019-10-31 12:20:52,681 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:21:09,467 INFO scheduling health checks: 2019-10-31 12:21:09,497 INFO scheduling jobs: 2019-10-31 12:21:09,499 DEBUG - qemu 2019-10-31 12:21:09,570 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:21:09,571 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:21:09,575 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:21:10,755 INFO [INIT] Dropping privileges 2019-10-31 12:21:10,755 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:21:10,755 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:21:10,757 INFO [INIT] Marking all workers as offline 2019-10-31 12:21:10,799 INFO [INIT] LAVA master has started. 2019-10-31 12:21:10,799 INFO [INIT] Using protocol version 3 2019-10-31 12:21:10,922 DEBUG lava-logs => PING(20) 2019-10-31 12:21:12,702 DEBUG Dhanu => PING(20) 2019-10-31 12:21:12,702 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:21:30,818 INFO scheduling health checks: 2019-10-31 12:21:30,843 INFO scheduling jobs: 2019-10-31 12:21:30,845 DEBUG - qemu 2019-10-31 12:21:30,919 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:21:30,919 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:21:30,924 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:21:32,088 INFO [INIT] Dropping privileges 2019-10-31 12:21:32,088 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:21:32,088 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:21:32,090 INFO [INIT] Marking all workers as offline 2019-10-31 12:21:32,123 INFO [INIT] LAVA master has started. 2019-10-31 12:21:32,124 INFO [INIT] Using protocol version 3 2019-10-31 12:21:32,200 DEBUG lava-logs => PING(20) 2019-10-31 12:21:32,721 DEBUG Dhanu => PING(20) 2019-10-31 12:21:32,721 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:21:52,144 INFO scheduling health checks: 2019-10-31 12:21:52,168 INFO scheduling jobs: 2019-10-31 12:21:52,169 DEBUG - qemu 2019-10-31 12:21:52,242 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:21:52,242 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:21:52,247 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:21:53,360 INFO [INIT] Dropping privileges 2019-10-31 12:21:53,361 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:21:53,361 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:21:53,362 INFO [INIT] Marking all workers as offline 2019-10-31 12:21:53,403 INFO [INIT] LAVA master has started. 2019-10-31 12:21:53,403 INFO [INIT] Using protocol version 3 2019-10-31 12:21:53,443 DEBUG Dhanu => PING(20) 2019-10-31 12:21:53,443 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:22:12,242 DEBUG lava-logs => PING(20) 2019-10-31 12:22:13,405 INFO scheduling health checks: 2019-10-31 12:22:13,425 INFO scheduling jobs: 2019-10-31 12:22:13,426 DEBUG - qemu 2019-10-31 12:22:13,498 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:22:13,498 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:22:13,502 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:22:14,595 INFO [INIT] Dropping privileges 2019-10-31 12:22:14,595 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:22:14,595 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:22:14,597 INFO [INIT] Marking all workers as offline 2019-10-31 12:22:14,628 INFO [INIT] LAVA master has started. 2019-10-31 12:22:14,628 INFO [INIT] Using protocol version 3 2019-10-31 12:22:32,263 DEBUG lava-logs => PING(20) 2019-10-31 12:22:33,488 DEBUG Dhanu => PING(20) 2019-10-31 12:22:33,488 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:22:34,629 INFO scheduling health checks: 2019-10-31 12:22:34,649 INFO scheduling jobs: 2019-10-31 12:22:34,650 DEBUG - qemu 2019-10-31 12:22:34,719 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:22:34,719 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:22:34,724 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:22:35,853 INFO [INIT] Dropping privileges 2019-10-31 12:22:35,854 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:22:35,854 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:22:35,855 INFO [INIT] Marking all workers as offline 2019-10-31 12:22:35,885 INFO [INIT] LAVA master has started. 2019-10-31 12:22:35,885 INFO [INIT] Using protocol version 3 2019-10-31 12:22:52,276 DEBUG lava-logs => PING(20) 2019-10-31 12:22:53,510 DEBUG Dhanu => PING(20) 2019-10-31 12:22:53,511 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:22:55,888 INFO scheduling health checks: 2019-10-31 12:22:55,907 INFO scheduling jobs: 2019-10-31 12:22:55,908 DEBUG - qemu 2019-10-31 12:22:55,978 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:22:55,978 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:22:55,983 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:22:57,086 INFO [INIT] Dropping privileges 2019-10-31 12:22:57,087 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:22:57,087 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:22:57,088 INFO [INIT] Marking all workers as offline 2019-10-31 12:22:57,121 INFO [INIT] LAVA master has started. 2019-10-31 12:22:57,121 INFO [INIT] Using protocol version 3 2019-10-31 12:23:12,294 DEBUG lava-logs => PING(20) 2019-10-31 12:23:13,530 DEBUG Dhanu => PING(20) 2019-10-31 12:23:13,531 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:23:17,125 INFO scheduling health checks: 2019-10-31 12:23:17,176 INFO scheduling jobs: 2019-10-31 12:23:17,180 DEBUG - qemu 2019-10-31 12:23:17,252 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:23:17,252 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:23:17,257 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:23:18,335 INFO [INIT] Dropping privileges 2019-10-31 12:23:18,335 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:23:18,335 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:23:18,337 INFO [INIT] Marking all workers as offline 2019-10-31 12:23:18,366 INFO [INIT] LAVA master has started. 2019-10-31 12:23:18,366 INFO [INIT] Using protocol version 3 2019-10-31 12:23:32,311 DEBUG lava-logs => PING(20) 2019-10-31 12:23:33,550 DEBUG Dhanu => PING(20) 2019-10-31 12:23:33,551 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:23:38,371 INFO scheduling health checks: 2019-10-31 12:23:38,400 INFO scheduling jobs: 2019-10-31 12:23:38,402 DEBUG - qemu 2019-10-31 12:23:38,475 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:23:38,475 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:23:38,480 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:23:39,619 INFO [INIT] Dropping privileges 2019-10-31 12:23:39,619 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:23:39,619 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:23:39,621 INFO [INIT] Marking all workers as offline 2019-10-31 12:23:39,658 INFO [INIT] LAVA master has started. 2019-10-31 12:23:39,659 INFO [INIT] Using protocol version 3 2019-10-31 12:23:52,326 DEBUG lava-logs => PING(20) 2019-10-31 12:23:53,573 DEBUG Dhanu => PING(20) 2019-10-31 12:23:53,573 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:23:59,665 INFO scheduling health checks: 2019-10-31 12:23:59,702 INFO scheduling jobs: 2019-10-31 12:23:59,704 DEBUG - qemu 2019-10-31 12:23:59,779 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:23:59,779 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:23:59,784 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:24:00,848 INFO [INIT] Dropping privileges 2019-10-31 12:24:00,848 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:24:00,848 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:24:00,850 INFO [INIT] Marking all workers as offline 2019-10-31 12:24:00,882 INFO [INIT] LAVA master has started. 2019-10-31 12:24:00,883 INFO [INIT] Using protocol version 3 2019-10-31 12:24:12,340 DEBUG lava-logs => PING(20) 2019-10-31 12:24:13,593 DEBUG Dhanu => PING(20) 2019-10-31 12:24:13,594 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:24:20,891 INFO scheduling health checks: 2019-10-31 12:24:20,938 INFO scheduling jobs: 2019-10-31 12:24:20,940 DEBUG - qemu 2019-10-31 12:24:21,008 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:24:21,008 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:24:21,012 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:24:22,096 INFO [INIT] Dropping privileges 2019-10-31 12:24:22,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:24:22,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:24:22,098 INFO [INIT] Marking all workers as offline 2019-10-31 12:24:22,129 INFO [INIT] LAVA master has started. 2019-10-31 12:24:22,130 INFO [INIT] Using protocol version 3 2019-10-31 12:24:32,350 DEBUG lava-logs => PING(20) 2019-10-31 12:24:33,614 DEBUG Dhanu => PING(20) 2019-10-31 12:24:33,615 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:24:42,138 INFO scheduling health checks: 2019-10-31 12:24:42,173 INFO scheduling jobs: 2019-10-31 12:24:42,175 DEBUG - qemu 2019-10-31 12:24:42,241 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:24:42,242 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:24:42,246 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:24:43,367 INFO [INIT] Dropping privileges 2019-10-31 12:24:43,368 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:24:43,368 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:24:43,369 INFO [INIT] Marking all workers as offline 2019-10-31 12:24:43,443 INFO [INIT] LAVA master has started. 2019-10-31 12:24:43,443 INFO [INIT] Using protocol version 3 2019-10-31 12:24:52,372 DEBUG lava-logs => PING(20) 2019-10-31 12:24:53,633 DEBUG Dhanu => PING(20) 2019-10-31 12:24:53,634 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:25:03,453 INFO scheduling health checks: 2019-10-31 12:25:03,498 INFO scheduling jobs: 2019-10-31 12:25:03,501 DEBUG - qemu 2019-10-31 12:25:03,575 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:25:03,575 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:25:03,580 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:25:04,848 INFO [INIT] Dropping privileges 2019-10-31 12:25:04,848 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:25:04,848 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:25:04,849 INFO [INIT] Marking all workers as offline 2019-10-31 12:25:04,877 INFO [INIT] LAVA master has started. 2019-10-31 12:25:04,878 INFO [INIT] Using protocol version 3 2019-10-31 12:25:12,391 DEBUG lava-logs => PING(20) 2019-10-31 12:25:13,645 DEBUG Dhanu => PING(20) 2019-10-31 12:25:13,646 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:25:24,881 INFO scheduling health checks: 2019-10-31 12:25:24,967 INFO scheduling jobs: 2019-10-31 12:25:24,970 DEBUG - qemu 2019-10-31 12:25:25,047 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:25:25,048 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:25:25,052 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:25:26,474 INFO [INIT] Dropping privileges 2019-10-31 12:25:26,474 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:25:26,474 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:25:26,475 INFO [INIT] Marking all workers as offline 2019-10-31 12:25:26,514 INFO [INIT] LAVA master has started. 2019-10-31 12:25:26,514 INFO [INIT] Using protocol version 3 2019-10-31 12:25:32,406 DEBUG lava-logs => PING(20) 2019-10-31 12:25:33,664 DEBUG Dhanu => PING(20) 2019-10-31 12:25:33,664 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:25:46,528 INFO scheduling health checks: 2019-10-31 12:25:46,568 INFO scheduling jobs: 2019-10-31 12:25:46,571 DEBUG - qemu 2019-10-31 12:25:46,644 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:25:46,644 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:25:46,649 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:25:47,863 INFO [INIT] Dropping privileges 2019-10-31 12:25:47,863 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:25:47,863 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:25:47,864 INFO [INIT] Marking all workers as offline 2019-10-31 12:25:47,903 INFO [INIT] LAVA master has started. 2019-10-31 12:25:47,904 INFO [INIT] Using protocol version 3 2019-10-31 12:25:52,423 DEBUG lava-logs => PING(20) 2019-10-31 12:25:53,678 DEBUG Dhanu => PING(20) 2019-10-31 12:25:53,679 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:26:07,917 INFO scheduling health checks: 2019-10-31 12:26:07,948 INFO scheduling jobs: 2019-10-31 12:26:07,950 DEBUG - qemu 2019-10-31 12:26:08,020 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:26:08,020 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:26:08,025 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:26:09,583 INFO [INIT] Dropping privileges 2019-10-31 12:26:09,583 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:26:09,583 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:26:09,585 INFO [INIT] Marking all workers as offline 2019-10-31 12:26:09,628 INFO [INIT] LAVA master has started. 2019-10-31 12:26:09,628 INFO [INIT] Using protocol version 3 2019-10-31 12:26:12,437 DEBUG lava-logs => PING(20) 2019-10-31 12:26:13,698 DEBUG Dhanu => PING(20) 2019-10-31 12:26:13,699 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:26:29,642 INFO scheduling health checks: 2019-10-31 12:26:29,661 INFO scheduling jobs: 2019-10-31 12:26:29,663 DEBUG - qemu 2019-10-31 12:26:29,736 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:26:29,736 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:26:29,741 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:26:31,180 INFO [INIT] Dropping privileges 2019-10-31 12:26:31,180 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:26:31,181 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:26:31,182 INFO [INIT] Marking all workers as offline 2019-10-31 12:26:31,220 INFO [INIT] LAVA master has started. 2019-10-31 12:26:31,221 INFO [INIT] Using protocol version 3 2019-10-31 12:26:32,448 DEBUG lava-logs => PING(20) 2019-10-31 12:26:33,713 DEBUG Dhanu => PING(20) 2019-10-31 12:26:33,714 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:26:51,239 INFO scheduling health checks: 2019-10-31 12:26:51,275 INFO scheduling jobs: 2019-10-31 12:26:51,277 DEBUG - qemu 2019-10-31 12:26:51,346 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:26:51,347 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:26:51,351 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:26:52,628 INFO [INIT] Dropping privileges 2019-10-31 12:26:52,628 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:26:52,628 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:26:52,630 INFO [INIT] Marking all workers as offline 2019-10-31 12:26:52,700 INFO [INIT] LAVA master has started. 2019-10-31 12:26:52,701 INFO [INIT] Using protocol version 3 2019-10-31 12:26:52,822 DEBUG lava-logs => PING(20) 2019-10-31 12:26:53,726 DEBUG Dhanu => PING(20) 2019-10-31 12:26:53,727 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:27:12,717 INFO scheduling health checks: 2019-10-31 12:27:12,766 INFO scheduling jobs: 2019-10-31 12:27:12,769 DEBUG - qemu 2019-10-31 12:27:12,847 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:27:12,848 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:27:12,855 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:27:14,265 INFO [INIT] Dropping privileges 2019-10-31 12:27:14,265 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:27:14,266 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:27:14,267 INFO [INIT] Marking all workers as offline 2019-10-31 12:27:14,313 INFO [INIT] LAVA master has started. 2019-10-31 12:27:14,313 INFO [INIT] Using protocol version 3 2019-10-31 12:27:14,327 DEBUG Dhanu => PING(20) 2019-10-31 12:27:14,328 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:27:32,859 DEBUG lava-logs => PING(20) 2019-10-31 12:27:34,315 INFO scheduling health checks: 2019-10-31 12:27:34,345 INFO scheduling jobs: 2019-10-31 12:27:34,347 DEBUG - qemu 2019-10-31 12:27:34,458 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:27:34,458 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:27:34,467 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:27:35,956 INFO [INIT] Dropping privileges 2019-10-31 12:27:35,956 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:27:35,956 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:27:35,957 INFO [INIT] Marking all workers as offline 2019-10-31 12:27:35,994 INFO [INIT] LAVA master has started. 2019-10-31 12:27:35,994 INFO [INIT] Using protocol version 3 2019-10-31 12:27:52,878 DEBUG lava-logs => PING(20) 2019-10-31 12:27:54,362 DEBUG Dhanu => PING(20) 2019-10-31 12:27:54,363 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:27:55,997 INFO scheduling health checks: 2019-10-31 12:27:56,092 INFO scheduling jobs: 2019-10-31 12:27:56,094 DEBUG - qemu 2019-10-31 12:27:56,162 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:27:56,162 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:27:56,167 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:27:57,513 INFO [INIT] Dropping privileges 2019-10-31 12:27:57,513 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:27:57,513 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:27:57,514 INFO [INIT] Marking all workers as offline 2019-10-31 12:27:57,550 INFO [INIT] LAVA master has started. 2019-10-31 12:27:57,550 INFO [INIT] Using protocol version 3 2019-10-31 12:28:12,893 DEBUG lava-logs => PING(20) 2019-10-31 12:28:14,378 DEBUG Dhanu => PING(20) 2019-10-31 12:28:14,378 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:28:17,553 INFO scheduling health checks: 2019-10-31 12:28:17,583 INFO scheduling jobs: 2019-10-31 12:28:17,586 DEBUG - qemu 2019-10-31 12:28:17,665 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:28:17,666 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:28:17,675 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:28:18,992 INFO [INIT] Dropping privileges 2019-10-31 12:28:18,992 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:28:18,992 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:28:18,994 INFO [INIT] Marking all workers as offline 2019-10-31 12:28:19,030 INFO [INIT] LAVA master has started. 2019-10-31 12:28:19,030 INFO [INIT] Using protocol version 3 2019-10-31 12:28:32,906 DEBUG lava-logs => PING(20) 2019-10-31 12:28:34,394 DEBUG Dhanu => PING(20) 2019-10-31 12:28:34,395 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:28:39,033 INFO scheduling health checks: 2019-10-31 12:28:39,063 INFO scheduling jobs: 2019-10-31 12:28:39,065 DEBUG - qemu 2019-10-31 12:28:39,148 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:28:39,149 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:28:39,155 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:28:40,382 INFO [INIT] Dropping privileges 2019-10-31 12:28:40,382 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:28:40,382 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:28:40,384 INFO [INIT] Marking all workers as offline 2019-10-31 12:28:40,421 INFO [INIT] LAVA master has started. 2019-10-31 12:28:40,422 INFO [INIT] Using protocol version 3 2019-10-31 12:28:52,922 DEBUG lava-logs => PING(20) 2019-10-31 12:28:54,413 DEBUG Dhanu => PING(20) 2019-10-31 12:28:54,414 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:29:00,428 INFO scheduling health checks: 2019-10-31 12:29:00,477 INFO scheduling jobs: 2019-10-31 12:29:00,478 DEBUG - qemu 2019-10-31 12:29:00,546 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:29:00,546 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:29:00,550 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:29:01,831 INFO [INIT] Dropping privileges 2019-10-31 12:29:01,831 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:29:01,831 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:29:01,833 INFO [INIT] Marking all workers as offline 2019-10-31 12:29:01,867 INFO [INIT] LAVA master has started. 2019-10-31 12:29:01,868 INFO [INIT] Using protocol version 3 2019-10-31 12:29:12,935 DEBUG lava-logs => PING(20) 2019-10-31 12:29:14,434 DEBUG Dhanu => PING(20) 2019-10-31 12:29:14,435 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:29:21,875 INFO scheduling health checks: 2019-10-31 12:29:21,912 INFO scheduling jobs: 2019-10-31 12:29:21,915 DEBUG - qemu 2019-10-31 12:29:21,988 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:29:21,988 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:29:21,993 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:29:23,091 INFO [INIT] Dropping privileges 2019-10-31 12:29:23,091 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:29:23,091 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:29:23,093 INFO [INIT] Marking all workers as offline 2019-10-31 12:29:23,160 INFO [INIT] LAVA master has started. 2019-10-31 12:29:23,160 INFO [INIT] Using protocol version 3 2019-10-31 12:29:32,946 DEBUG lava-logs => PING(20) 2019-10-31 12:29:34,452 DEBUG Dhanu => PING(20) 2019-10-31 12:29:34,452 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:29:43,169 INFO scheduling health checks: 2019-10-31 12:29:43,203 INFO scheduling jobs: 2019-10-31 12:29:43,204 DEBUG - qemu 2019-10-31 12:29:43,298 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:29:43,298 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:29:43,303 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:29:44,733 INFO [INIT] Dropping privileges 2019-10-31 12:29:44,733 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:29:44,733 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:29:44,735 INFO [INIT] Marking all workers as offline 2019-10-31 12:29:44,794 INFO [INIT] LAVA master has started. 2019-10-31 12:29:44,794 INFO [INIT] Using protocol version 3 2019-10-31 12:29:52,962 DEBUG lava-logs => PING(20) 2019-10-31 12:29:54,470 DEBUG Dhanu => PING(20) 2019-10-31 12:29:54,470 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:30:04,805 INFO scheduling health checks: 2019-10-31 12:30:04,850 INFO scheduling jobs: 2019-10-31 12:30:04,852 DEBUG - qemu 2019-10-31 12:30:04,926 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:30:04,927 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:30:04,942 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:30:06,120 INFO [INIT] Dropping privileges 2019-10-31 12:30:06,120 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:30:06,120 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:30:06,122 INFO [INIT] Marking all workers as offline 2019-10-31 12:30:06,186 INFO [INIT] LAVA master has started. 2019-10-31 12:30:06,187 INFO [INIT] Using protocol version 3 2019-10-31 12:30:12,977 DEBUG lava-logs => PING(20) 2019-10-31 12:30:14,492 DEBUG Dhanu => PING(20) 2019-10-31 12:30:14,492 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:30:26,199 INFO scheduling health checks: 2019-10-31 12:30:26,241 INFO scheduling jobs: 2019-10-31 12:30:26,243 DEBUG - qemu 2019-10-31 12:30:26,316 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:30:26,317 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:30:26,321 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:30:27,614 INFO [INIT] Dropping privileges 2019-10-31 12:30:27,614 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:30:27,614 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:30:27,616 INFO [INIT] Marking all workers as offline 2019-10-31 12:30:27,653 INFO [INIT] LAVA master has started. 2019-10-31 12:30:27,653 INFO [INIT] Using protocol version 3 2019-10-31 12:30:32,990 DEBUG lava-logs => PING(20) 2019-10-31 12:30:34,513 DEBUG Dhanu => PING(20) 2019-10-31 12:30:34,514 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:30:47,667 INFO scheduling health checks: 2019-10-31 12:30:47,698 INFO scheduling jobs: 2019-10-31 12:30:47,700 DEBUG - qemu 2019-10-31 12:30:47,770 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:30:47,770 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:30:47,774 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:30:48,856 INFO [INIT] Dropping privileges 2019-10-31 12:30:48,856 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:30:48,857 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:30:48,858 INFO [INIT] Marking all workers as offline 2019-10-31 12:30:48,890 INFO [INIT] LAVA master has started. 2019-10-31 12:30:48,891 INFO [INIT] Using protocol version 3 2019-10-31 12:30:53,006 DEBUG lava-logs => PING(20) 2019-10-31 12:30:54,534 DEBUG Dhanu => PING(20) 2019-10-31 12:30:54,534 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:31:08,905 INFO scheduling health checks: 2019-10-31 12:31:08,945 INFO scheduling jobs: 2019-10-31 12:31:08,947 DEBUG - qemu 2019-10-31 12:31:09,013 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:31:09,013 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:31:09,028 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:31:10,091 INFO [INIT] Dropping privileges 2019-10-31 12:31:10,091 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:31:10,091 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:31:10,093 INFO [INIT] Marking all workers as offline 2019-10-31 12:31:10,124 INFO [INIT] LAVA master has started. 2019-10-31 12:31:10,124 INFO [INIT] Using protocol version 3 2019-10-31 12:31:13,022 DEBUG lava-logs => PING(20) 2019-10-31 12:31:14,555 DEBUG Dhanu => PING(20) 2019-10-31 12:31:14,556 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:31:30,140 INFO scheduling health checks: 2019-10-31 12:31:30,192 INFO scheduling jobs: 2019-10-31 12:31:30,195 DEBUG - qemu 2019-10-31 12:31:30,266 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:31:30,267 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:31:30,271 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:31:31,339 INFO [INIT] Dropping privileges 2019-10-31 12:31:31,339 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:31:31,339 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:31:31,341 INFO [INIT] Marking all workers as offline 2019-10-31 12:31:31,382 INFO [INIT] LAVA master has started. 2019-10-31 12:31:31,382 INFO [INIT] Using protocol version 3 2019-10-31 12:31:33,037 DEBUG lava-logs => PING(20) 2019-10-31 12:31:34,574 DEBUG Dhanu => PING(20) 2019-10-31 12:31:34,574 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:31:51,387 INFO scheduling health checks: 2019-10-31 12:31:51,416 INFO scheduling jobs: 2019-10-31 12:31:51,418 DEBUG - qemu 2019-10-31 12:31:51,489 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:31:51,489 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:31:51,494 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:31:52,632 INFO [INIT] Dropping privileges 2019-10-31 12:31:52,632 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:31:52,632 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:31:52,634 INFO [INIT] Marking all workers as offline 2019-10-31 12:31:52,708 INFO [INIT] LAVA master has started. 2019-10-31 12:31:52,708 INFO [INIT] Using protocol version 3 2019-10-31 12:31:53,050 DEBUG lava-logs => PING(20) 2019-10-31 12:31:54,592 DEBUG Dhanu => PING(20) 2019-10-31 12:31:54,593 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:32:12,726 INFO scheduling health checks: 2019-10-31 12:32:12,751 INFO scheduling jobs: 2019-10-31 12:32:12,753 DEBUG - qemu 2019-10-31 12:32:12,822 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:32:12,823 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:32:12,827 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:32:14,086 INFO [INIT] Dropping privileges 2019-10-31 12:32:14,086 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:32:14,086 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:32:14,088 INFO [INIT] Marking all workers as offline 2019-10-31 12:32:14,119 INFO [INIT] LAVA master has started. 2019-10-31 12:32:14,120 INFO [INIT] Using protocol version 3 2019-10-31 12:32:14,144 DEBUG lava-logs => PING(20) 2019-10-31 12:32:14,610 DEBUG Dhanu => PING(20) 2019-10-31 12:32:14,611 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:32:34,140 INFO scheduling health checks: 2019-10-31 12:32:34,165 INFO scheduling jobs: 2019-10-31 12:32:34,166 DEBUG - qemu 2019-10-31 12:32:34,240 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:32:34,241 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:32:34,245 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:32:35,348 INFO [INIT] Dropping privileges 2019-10-31 12:32:35,349 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:32:35,349 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:32:35,350 INFO [INIT] Marking all workers as offline 2019-10-31 12:32:35,389 INFO [INIT] LAVA master has started. 2019-10-31 12:32:35,389 INFO [INIT] Using protocol version 3 2019-10-31 12:32:35,476 DEBUG Dhanu => PING(20) 2019-10-31 12:32:35,477 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:32:54,186 DEBUG lava-logs => PING(20) 2019-10-31 12:32:55,391 INFO scheduling health checks: 2019-10-31 12:32:55,411 INFO scheduling jobs: 2019-10-31 12:32:55,412 DEBUG - qemu 2019-10-31 12:32:55,483 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:32:55,483 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:32:55,487 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:32:56,605 INFO [INIT] Dropping privileges 2019-10-31 12:32:56,606 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:32:56,606 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:32:56,607 INFO [INIT] Marking all workers as offline 2019-10-31 12:32:56,646 INFO [INIT] LAVA master has started. 2019-10-31 12:32:56,647 INFO [INIT] Using protocol version 3 2019-10-31 12:32:56,706 DEBUG Dhanu => PING(20) 2019-10-31 12:32:56,707 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:33:14,206 DEBUG lava-logs => PING(20) 2019-10-31 12:33:16,650 INFO scheduling health checks: 2019-10-31 12:33:16,680 INFO scheduling jobs: 2019-10-31 12:33:16,682 DEBUG - qemu 2019-10-31 12:33:16,755 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:33:16,756 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:33:16,760 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:33:17,848 INFO [INIT] Dropping privileges 2019-10-31 12:33:17,848 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:33:17,848 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:33:17,850 INFO [INIT] Marking all workers as offline 2019-10-31 12:33:17,883 INFO [INIT] LAVA master has started. 2019-10-31 12:33:17,883 INFO [INIT] Using protocol version 3 2019-10-31 12:33:34,225 DEBUG lava-logs => PING(20) 2019-10-31 12:33:36,748 DEBUG Dhanu => PING(20) 2019-10-31 12:33:36,748 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:33:37,885 INFO scheduling health checks: 2019-10-31 12:33:37,915 INFO scheduling jobs: 2019-10-31 12:33:37,917 DEBUG - qemu 2019-10-31 12:33:37,990 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:33:37,990 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:33:37,994 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:33:39,096 INFO [INIT] Dropping privileges 2019-10-31 12:33:39,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:33:39,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:33:39,098 INFO [INIT] Marking all workers as offline 2019-10-31 12:33:39,139 INFO [INIT] LAVA master has started. 2019-10-31 12:33:39,139 INFO [INIT] Using protocol version 3 2019-10-31 12:33:54,241 DEBUG lava-logs => PING(20) 2019-10-31 12:33:56,770 DEBUG Dhanu => PING(20) 2019-10-31 12:33:56,771 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:33:59,141 INFO scheduling health checks: 2019-10-31 12:33:59,173 INFO scheduling jobs: 2019-10-31 12:33:59,175 DEBUG - qemu 2019-10-31 12:33:59,246 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:33:59,246 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:33:59,251 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:34:00,355 INFO [INIT] Dropping privileges 2019-10-31 12:34:00,355 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:34:00,356 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:34:00,357 INFO [INIT] Marking all workers as offline 2019-10-31 12:34:00,397 INFO [INIT] LAVA master has started. 2019-10-31 12:34:00,397 INFO [INIT] Using protocol version 3 2019-10-31 12:34:14,258 DEBUG lava-logs => PING(20) 2019-10-31 12:34:16,791 DEBUG Dhanu => PING(20) 2019-10-31 12:34:16,791 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:34:20,401 INFO scheduling health checks: 2019-10-31 12:34:20,431 INFO scheduling jobs: 2019-10-31 12:34:20,433 DEBUG - qemu 2019-10-31 12:34:20,505 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:34:20,506 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:34:20,510 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:34:21,587 INFO [INIT] Dropping privileges 2019-10-31 12:34:21,588 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:34:21,588 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:34:21,589 INFO [INIT] Marking all workers as offline 2019-10-31 12:34:21,621 INFO [INIT] LAVA master has started. 2019-10-31 12:34:21,621 INFO [INIT] Using protocol version 3 2019-10-31 12:34:34,271 DEBUG lava-logs => PING(20) 2019-10-31 12:34:36,808 DEBUG Dhanu => PING(20) 2019-10-31 12:34:36,809 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:34:41,627 INFO scheduling health checks: 2019-10-31 12:34:41,666 INFO scheduling jobs: 2019-10-31 12:34:41,669 DEBUG - qemu 2019-10-31 12:34:41,744 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:34:41,744 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:34:41,749 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:34:42,835 INFO [INIT] Dropping privileges 2019-10-31 12:34:42,835 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:34:42,835 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:34:42,837 INFO [INIT] Marking all workers as offline 2019-10-31 12:34:42,880 INFO [INIT] LAVA master has started. 2019-10-31 12:34:42,880 INFO [INIT] Using protocol version 3 2019-10-31 12:34:54,286 DEBUG lava-logs => PING(20) 2019-10-31 12:34:56,830 DEBUG Dhanu => PING(20) 2019-10-31 12:34:56,831 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:35:02,885 INFO scheduling health checks: 2019-10-31 12:35:02,924 INFO scheduling jobs: 2019-10-31 12:35:02,927 DEBUG - qemu 2019-10-31 12:35:03,000 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:35:03,000 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:35:03,005 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:35:04,111 INFO [INIT] Dropping privileges 2019-10-31 12:35:04,111 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:35:04,111 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:35:04,113 INFO [INIT] Marking all workers as offline 2019-10-31 12:35:04,171 INFO [INIT] LAVA master has started. 2019-10-31 12:35:04,172 INFO [INIT] Using protocol version 3 2019-10-31 12:35:14,299 DEBUG lava-logs => PING(20) 2019-10-31 12:35:16,850 DEBUG Dhanu => PING(20) 2019-10-31 12:35:16,851 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:35:24,180 INFO scheduling health checks: 2019-10-31 12:35:24,215 INFO scheduling jobs: 2019-10-31 12:35:24,217 DEBUG - qemu 2019-10-31 12:35:24,294 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:35:24,295 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:35:24,299 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:35:25,624 INFO [INIT] Dropping privileges 2019-10-31 12:35:25,624 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:35:25,624 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:35:25,626 INFO [INIT] Marking all workers as offline 2019-10-31 12:35:25,662 INFO [INIT] LAVA master has started. 2019-10-31 12:35:25,662 INFO [INIT] Using protocol version 3 2019-10-31 12:35:34,318 DEBUG lava-logs => PING(20) 2019-10-31 12:35:36,870 DEBUG Dhanu => PING(20) 2019-10-31 12:35:36,871 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:35:45,671 INFO scheduling health checks: 2019-10-31 12:35:45,717 INFO scheduling jobs: 2019-10-31 12:35:45,719 DEBUG - qemu 2019-10-31 12:35:45,793 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:35:45,793 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:35:45,798 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:35:47,096 INFO [INIT] Dropping privileges 2019-10-31 12:35:47,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:35:47,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:35:47,098 INFO [INIT] Marking all workers as offline 2019-10-31 12:35:47,130 INFO [INIT] LAVA master has started. 2019-10-31 12:35:47,130 INFO [INIT] Using protocol version 3 2019-10-31 12:35:54,338 DEBUG lava-logs => PING(20) 2019-10-31 12:35:56,890 DEBUG Dhanu => PING(20) 2019-10-31 12:35:56,890 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:36:07,137 INFO scheduling health checks: 2019-10-31 12:36:07,174 INFO scheduling jobs: 2019-10-31 12:36:07,177 DEBUG - qemu 2019-10-31 12:36:07,263 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:36:07,263 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:36:07,268 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:36:08,537 INFO [INIT] Dropping privileges 2019-10-31 12:36:08,537 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:36:08,537 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:36:08,539 INFO [INIT] Marking all workers as offline 2019-10-31 12:36:08,565 INFO [INIT] LAVA master has started. 2019-10-31 12:36:08,565 INFO [INIT] Using protocol version 3 2019-10-31 12:36:14,353 DEBUG lava-logs => PING(20) 2019-10-31 12:36:16,912 DEBUG Dhanu => PING(20) 2019-10-31 12:36:16,912 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:36:28,577 INFO scheduling health checks: 2019-10-31 12:36:28,610 INFO scheduling jobs: 2019-10-31 12:36:28,614 DEBUG - qemu 2019-10-31 12:36:28,690 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:36:28,690 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:36:28,699 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:36:29,892 INFO [INIT] Dropping privileges 2019-10-31 12:36:29,893 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:36:29,893 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:36:29,894 INFO [INIT] Marking all workers as offline 2019-10-31 12:36:29,924 INFO [INIT] LAVA master has started. 2019-10-31 12:36:29,924 INFO [INIT] Using protocol version 3 2019-10-31 12:36:34,368 DEBUG lava-logs => PING(20) 2019-10-31 12:36:36,931 DEBUG Dhanu => PING(20) 2019-10-31 12:36:36,931 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:36:49,934 INFO scheduling health checks: 2019-10-31 12:36:49,968 INFO scheduling jobs: 2019-10-31 12:36:49,970 DEBUG - qemu 2019-10-31 12:36:50,049 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:36:50,049 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:36:50,054 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:36:51,493 INFO [INIT] Dropping privileges 2019-10-31 12:36:51,494 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:36:51,494 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:36:51,495 INFO [INIT] Marking all workers as offline 2019-10-31 12:36:51,560 INFO [INIT] LAVA master has started. 2019-10-31 12:36:51,560 INFO [INIT] Using protocol version 3 2019-10-31 12:36:54,381 DEBUG lava-logs => PING(20) 2019-10-31 12:36:56,946 DEBUG Dhanu => PING(20) 2019-10-31 12:36:56,947 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:37:11,573 INFO scheduling health checks: 2019-10-31 12:37:11,603 INFO scheduling jobs: 2019-10-31 12:37:11,605 DEBUG - qemu 2019-10-31 12:37:11,685 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:37:11,685 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:37:11,689 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:37:12,920 INFO [INIT] Dropping privileges 2019-10-31 12:37:12,920 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:37:12,920 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:37:12,922 INFO [INIT] Marking all workers as offline 2019-10-31 12:37:12,961 INFO [INIT] LAVA master has started. 2019-10-31 12:37:12,961 INFO [INIT] Using protocol version 3 2019-10-31 12:37:14,393 DEBUG lava-logs => PING(20) 2019-10-31 12:37:16,968 DEBUG Dhanu => PING(20) 2019-10-31 12:37:16,968 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:37:32,977 INFO scheduling health checks: 2019-10-31 12:37:33,027 INFO scheduling jobs: 2019-10-31 12:37:33,031 DEBUG - qemu 2019-10-31 12:37:33,113 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:37:33,114 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:37:33,122 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:37:34,498 INFO [INIT] Dropping privileges 2019-10-31 12:37:34,498 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:37:34,498 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:37:34,500 INFO [INIT] Marking all workers as offline 2019-10-31 12:37:34,529 INFO [INIT] LAVA master has started. 2019-10-31 12:37:34,529 INFO [INIT] Using protocol version 3 2019-10-31 12:37:34,575 DEBUG lava-logs => PING(20) 2019-10-31 12:37:36,985 DEBUG Dhanu => PING(20) 2019-10-31 12:37:36,986 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:37:54,547 INFO scheduling health checks: 2019-10-31 12:37:54,585 INFO scheduling jobs: 2019-10-31 12:37:54,587 DEBUG - qemu 2019-10-31 12:37:54,669 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:37:54,669 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:37:54,673 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:37:55,889 INFO [INIT] Dropping privileges 2019-10-31 12:37:55,890 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:37:55,890 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:37:55,891 INFO [INIT] Marking all workers as offline 2019-10-31 12:37:55,931 INFO [INIT] LAVA master has started. 2019-10-31 12:37:55,931 INFO [INIT] Using protocol version 3 2019-10-31 12:37:56,999 DEBUG Dhanu => PING(20) 2019-10-31 12:37:56,999 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:38:14,614 DEBUG lava-logs => PING(20) 2019-10-31 12:38:15,933 INFO scheduling health checks: 2019-10-31 12:38:15,954 INFO scheduling jobs: 2019-10-31 12:38:15,955 DEBUG - qemu 2019-10-31 12:38:16,050 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:38:16,050 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:38:16,058 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:38:17,543 INFO [INIT] Dropping privileges 2019-10-31 12:38:17,544 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:38:17,544 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:38:17,545 INFO [INIT] Marking all workers as offline 2019-10-31 12:38:17,578 INFO [INIT] LAVA master has started. 2019-10-31 12:38:17,578 INFO [INIT] Using protocol version 3 2019-10-31 12:38:17,704 DEBUG Dhanu => PING(20) 2019-10-31 12:38:17,704 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:38:34,630 DEBUG lava-logs => PING(20) 2019-10-31 12:38:37,581 INFO scheduling health checks: 2019-10-31 12:38:37,611 INFO scheduling jobs: 2019-10-31 12:38:37,613 DEBUG - qemu 2019-10-31 12:38:37,693 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:38:37,694 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:38:37,699 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:38:39,122 INFO [INIT] Dropping privileges 2019-10-31 12:38:39,123 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:38:39,123 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:38:39,124 INFO [INIT] Marking all workers as offline 2019-10-31 12:38:39,205 INFO [INIT] LAVA master has started. 2019-10-31 12:38:39,206 INFO [INIT] Using protocol version 3 2019-10-31 12:38:39,272 DEBUG Dhanu => PING(20) 2019-10-31 12:38:39,272 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:38:54,642 DEBUG lava-logs => PING(20) 2019-10-31 12:38:59,209 INFO scheduling health checks: 2019-10-31 12:38:59,263 INFO scheduling jobs: 2019-10-31 12:38:59,266 DEBUG - qemu 2019-10-31 12:38:59,343 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:38:59,343 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:38:59,350 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:39:00,734 INFO [INIT] Dropping privileges 2019-10-31 12:39:00,734 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:39:00,734 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:39:00,736 INFO [INIT] Marking all workers as offline 2019-10-31 12:39:00,794 INFO [INIT] LAVA master has started. 2019-10-31 12:39:00,795 INFO [INIT] Using protocol version 3 2019-10-31 12:39:14,656 DEBUG lava-logs => PING(20) 2019-10-31 12:39:19,306 DEBUG Dhanu => PING(20) 2019-10-31 12:39:19,307 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:39:20,796 INFO scheduling health checks: 2019-10-31 12:39:20,827 INFO scheduling jobs: 2019-10-31 12:39:20,829 DEBUG - qemu 2019-10-31 12:39:20,904 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:39:20,904 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:39:20,909 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:39:22,123 INFO [INIT] Dropping privileges 2019-10-31 12:39:22,123 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:39:22,123 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:39:22,125 INFO [INIT] Marking all workers as offline 2019-10-31 12:39:22,183 INFO [INIT] LAVA master has started. 2019-10-31 12:39:22,183 INFO [INIT] Using protocol version 3 2019-10-31 12:39:34,670 DEBUG lava-logs => PING(20) 2019-10-31 12:39:39,326 DEBUG Dhanu => PING(20) 2019-10-31 12:39:39,327 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:39:42,185 INFO scheduling health checks: 2019-10-31 12:39:42,207 INFO scheduling jobs: 2019-10-31 12:39:42,208 DEBUG - qemu 2019-10-31 12:39:42,280 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:39:42,280 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:39:42,285 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:39:43,662 INFO [INIT] Dropping privileges 2019-10-31 12:39:43,663 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:39:43,663 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:39:43,664 INFO [INIT] Marking all workers as offline 2019-10-31 12:39:43,698 INFO [INIT] LAVA master has started. 2019-10-31 12:39:43,698 INFO [INIT] Using protocol version 3 2019-10-31 12:39:54,684 DEBUG lava-logs => PING(20) 2019-10-31 12:39:59,344 DEBUG Dhanu => PING(20) 2019-10-31 12:39:59,344 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:40:03,703 INFO scheduling health checks: 2019-10-31 12:40:03,742 INFO scheduling jobs: 2019-10-31 12:40:03,745 DEBUG - qemu 2019-10-31 12:40:03,833 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:40:03,833 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:40:03,838 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:40:05,186 INFO [INIT] Dropping privileges 2019-10-31 12:40:05,186 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:40:05,186 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:40:05,188 INFO [INIT] Marking all workers as offline 2019-10-31 12:40:05,223 INFO [INIT] LAVA master has started. 2019-10-31 12:40:05,223 INFO [INIT] Using protocol version 3 2019-10-31 12:40:14,702 DEBUG lava-logs => PING(20) 2019-10-31 12:40:19,362 DEBUG Dhanu => PING(20) 2019-10-31 12:40:19,363 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:40:25,229 INFO scheduling health checks: 2019-10-31 12:40:25,255 INFO scheduling jobs: 2019-10-31 12:40:25,257 DEBUG - qemu 2019-10-31 12:40:25,333 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:40:25,333 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:40:25,338 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:40:26,705 INFO [INIT] Dropping privileges 2019-10-31 12:40:26,706 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:40:26,706 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:40:26,707 INFO [INIT] Marking all workers as offline 2019-10-31 12:40:26,748 INFO [INIT] LAVA master has started. 2019-10-31 12:40:26,748 INFO [INIT] Using protocol version 3 2019-10-31 12:40:34,723 DEBUG lava-logs => PING(20) 2019-10-31 12:40:39,378 DEBUG Dhanu => PING(20) 2019-10-31 12:40:39,378 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:40:46,753 INFO scheduling health checks: 2019-10-31 12:40:46,791 INFO scheduling jobs: 2019-10-31 12:40:46,794 DEBUG - qemu 2019-10-31 12:40:46,870 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:40:46,870 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:40:46,875 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:40:48,358 INFO [INIT] Dropping privileges 2019-10-31 12:40:48,358 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:40:48,359 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:40:48,361 INFO [INIT] Marking all workers as offline 2019-10-31 12:40:48,438 INFO [INIT] LAVA master has started. 2019-10-31 12:40:48,438 INFO [INIT] Using protocol version 3 2019-10-31 12:40:54,738 DEBUG lava-logs => PING(20) 2019-10-31 12:40:59,400 DEBUG Dhanu => PING(20) 2019-10-31 12:40:59,400 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:41:08,445 INFO scheduling health checks: 2019-10-31 12:41:08,482 INFO scheduling jobs: 2019-10-31 12:41:08,484 DEBUG - qemu 2019-10-31 12:41:08,565 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:41:08,565 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:41:08,570 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:41:09,973 INFO [INIT] Dropping privileges 2019-10-31 12:41:09,973 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:41:09,973 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:41:09,975 INFO [INIT] Marking all workers as offline 2019-10-31 12:41:10,007 INFO [INIT] LAVA master has started. 2019-10-31 12:41:10,007 INFO [INIT] Using protocol version 3 2019-10-31 12:41:14,750 DEBUG lava-logs => PING(20) 2019-10-31 12:41:19,421 DEBUG Dhanu => PING(20) 2019-10-31 12:41:19,421 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:41:30,018 INFO scheduling health checks: 2019-10-31 12:41:30,051 INFO scheduling jobs: 2019-10-31 12:41:30,052 DEBUG - qemu 2019-10-31 12:41:30,119 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:41:30,119 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:41:30,124 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:41:31,354 INFO [INIT] Dropping privileges 2019-10-31 12:41:31,355 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:41:31,355 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:41:31,356 INFO [INIT] Marking all workers as offline 2019-10-31 12:41:31,397 INFO [INIT] LAVA master has started. 2019-10-31 12:41:31,397 INFO [INIT] Using protocol version 3 2019-10-31 12:41:34,762 DEBUG lava-logs => PING(20) 2019-10-31 12:41:39,440 DEBUG Dhanu => PING(20) 2019-10-31 12:41:39,440 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:41:51,405 INFO scheduling health checks: 2019-10-31 12:41:51,441 INFO scheduling jobs: 2019-10-31 12:41:51,444 DEBUG - qemu 2019-10-31 12:41:51,532 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:41:51,532 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:41:51,538 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:41:53,061 INFO [INIT] Dropping privileges 2019-10-31 12:41:53,062 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:41:53,062 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:41:53,063 INFO [INIT] Marking all workers as offline 2019-10-31 12:41:53,100 INFO [INIT] LAVA master has started. 2019-10-31 12:41:53,100 INFO [INIT] Using protocol version 3 2019-10-31 12:41:54,776 DEBUG lava-logs => PING(20) 2019-10-31 12:41:59,460 DEBUG Dhanu => PING(20) 2019-10-31 12:41:59,461 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:42:13,113 INFO scheduling health checks: 2019-10-31 12:42:13,156 INFO scheduling jobs: 2019-10-31 12:42:13,159 DEBUG - qemu 2019-10-31 12:42:13,250 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:42:13,252 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:42:13,257 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:42:14,541 INFO [INIT] Dropping privileges 2019-10-31 12:42:14,541 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:42:14,542 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:42:14,543 INFO [INIT] Marking all workers as offline 2019-10-31 12:42:14,578 INFO [INIT] LAVA master has started. 2019-10-31 12:42:14,578 INFO [INIT] Using protocol version 3 2019-10-31 12:42:14,791 DEBUG lava-logs => PING(20) 2019-10-31 12:42:19,478 DEBUG Dhanu => PING(20) 2019-10-31 12:42:19,479 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:42:34,593 INFO scheduling health checks: 2019-10-31 12:42:34,624 INFO scheduling jobs: 2019-10-31 12:42:34,628 DEBUG - qemu 2019-10-31 12:42:34,716 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:42:34,717 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:42:34,726 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:42:36,151 INFO [INIT] Dropping privileges 2019-10-31 12:42:36,152 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:42:36,152 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:42:36,156 INFO [INIT] Marking all workers as offline 2019-10-31 12:42:36,238 INFO [INIT] LAVA master has started. 2019-10-31 12:42:36,238 INFO [INIT] Using protocol version 3 2019-10-31 12:42:36,397 DEBUG lava-logs => PING(20) 2019-10-31 12:42:39,493 DEBUG Dhanu => PING(20) 2019-10-31 12:42:39,493 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:42:56,253 INFO scheduling health checks: 2019-10-31 12:42:56,281 INFO scheduling jobs: 2019-10-31 12:42:56,285 DEBUG - qemu 2019-10-31 12:42:56,359 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:42:56,360 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:42:56,364 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:42:57,675 INFO [INIT] Dropping privileges 2019-10-31 12:42:57,675 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:42:57,676 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:42:57,677 INFO [INIT] Marking all workers as offline 2019-10-31 12:42:57,718 INFO [INIT] LAVA master has started. 2019-10-31 12:42:57,718 INFO [INIT] Using protocol version 3 2019-10-31 12:42:57,776 DEBUG lava-logs => PING(20) 2019-10-31 12:42:59,508 DEBUG Dhanu => PING(20) 2019-10-31 12:42:59,509 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:43:17,734 INFO scheduling health checks: 2019-10-31 12:43:17,761 INFO scheduling jobs: 2019-10-31 12:43:17,763 DEBUG - qemu 2019-10-31 12:43:17,840 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:43:17,840 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:43:17,845 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:43:19,119 INFO [INIT] Dropping privileges 2019-10-31 12:43:19,120 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:43:19,120 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:43:19,121 INFO [INIT] Marking all workers as offline 2019-10-31 12:43:19,164 INFO [INIT] LAVA master has started. 2019-10-31 12:43:19,165 INFO [INIT] Using protocol version 3 2019-10-31 12:43:19,526 DEBUG Dhanu => PING(20) 2019-10-31 12:43:19,527 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:43:37,813 DEBUG lava-logs => PING(20) 2019-10-31 12:43:39,166 INFO scheduling health checks: 2019-10-31 12:43:39,275 INFO scheduling jobs: 2019-10-31 12:43:39,277 DEBUG - qemu 2019-10-31 12:43:39,400 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:43:39,400 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:43:39,407 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:43:40,904 INFO [INIT] Dropping privileges 2019-10-31 12:43:40,904 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:43:40,904 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:43:40,906 INFO [INIT] Marking all workers as offline 2019-10-31 12:43:40,982 INFO [INIT] LAVA master has started. 2019-10-31 12:43:40,982 INFO [INIT] Using protocol version 3 2019-10-31 12:43:41,129 DEBUG Dhanu => PING(20) 2019-10-31 12:43:41,129 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:43:57,831 DEBUG lava-logs => PING(20) 2019-10-31 12:44:00,986 INFO scheduling health checks: 2019-10-31 12:44:01,044 INFO scheduling jobs: 2019-10-31 12:44:01,047 DEBUG - qemu 2019-10-31 12:44:01,120 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:44:01,120 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:44:01,124 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:44:02,345 INFO [INIT] Dropping privileges 2019-10-31 12:44:02,345 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:44:02,345 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:44:02,347 INFO [INIT] Marking all workers as offline 2019-10-31 12:44:02,378 INFO [INIT] LAVA master has started. 2019-10-31 12:44:02,379 INFO [INIT] Using protocol version 3 2019-10-31 12:44:02,573 DEBUG Dhanu => PING(20) 2019-10-31 12:44:02,574 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:44:17,841 DEBUG lava-logs => PING(20) 2019-10-31 12:44:22,381 INFO scheduling health checks: 2019-10-31 12:44:22,412 INFO scheduling jobs: 2019-10-31 12:44:22,414 DEBUG - qemu 2019-10-31 12:44:22,483 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:44:22,483 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:44:22,488 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:44:23,700 INFO [INIT] Dropping privileges 2019-10-31 12:44:23,701 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:44:23,701 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:44:23,702 INFO [INIT] Marking all workers as offline 2019-10-31 12:44:23,763 INFO [INIT] LAVA master has started. 2019-10-31 12:44:23,763 INFO [INIT] Using protocol version 3 2019-10-31 12:44:23,875 DEBUG Dhanu => PING(20) 2019-10-31 12:44:23,875 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:44:37,853 DEBUG lava-logs => PING(20) 2019-10-31 12:44:43,769 INFO scheduling health checks: 2019-10-31 12:44:43,803 INFO scheduling jobs: 2019-10-31 12:44:43,804 DEBUG - qemu 2019-10-31 12:44:43,871 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:44:43,871 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:44:43,876 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:44:45,159 INFO [INIT] Dropping privileges 2019-10-31 12:44:45,159 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:44:45,160 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:44:45,161 INFO [INIT] Marking all workers as offline 2019-10-31 12:44:45,204 INFO [INIT] LAVA master has started. 2019-10-31 12:44:45,204 INFO [INIT] Using protocol version 3 2019-10-31 12:44:45,263 DEBUG Dhanu => PING(20) 2019-10-31 12:44:45,264 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:44:57,865 DEBUG lava-logs => PING(20) 2019-10-31 12:45:05,209 INFO scheduling health checks: 2019-10-31 12:45:05,238 INFO scheduling jobs: 2019-10-31 12:45:05,240 DEBUG - qemu 2019-10-31 12:45:05,308 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:45:05,309 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:45:05,314 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:45:06,678 INFO [INIT] Dropping privileges 2019-10-31 12:45:06,678 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:45:06,678 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:45:06,686 INFO [INIT] Marking all workers as offline 2019-10-31 12:45:06,995 INFO [INIT] LAVA master has started. 2019-10-31 12:45:06,996 INFO [INIT] Using protocol version 3 2019-10-31 12:45:17,878 DEBUG lava-logs => PING(20) 2019-10-31 12:45:25,299 DEBUG Dhanu => PING(20) 2019-10-31 12:45:25,299 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:45:26,997 INFO scheduling health checks: 2019-10-31 12:45:27,030 INFO scheduling jobs: 2019-10-31 12:45:27,031 DEBUG - qemu 2019-10-31 12:45:27,100 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:45:27,100 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:45:27,105 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:45:28,307 INFO [INIT] Dropping privileges 2019-10-31 12:45:28,307 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:45:28,307 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:45:28,309 INFO [INIT] Marking all workers as offline 2019-10-31 12:45:28,366 INFO [INIT] LAVA master has started. 2019-10-31 12:45:28,367 INFO [INIT] Using protocol version 3 2019-10-31 12:45:37,900 DEBUG lava-logs => PING(20) 2019-10-31 12:45:45,314 DEBUG Dhanu => PING(20) 2019-10-31 12:45:45,315 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:45:48,369 INFO scheduling health checks: 2019-10-31 12:45:48,421 INFO scheduling jobs: 2019-10-31 12:45:48,424 DEBUG - qemu 2019-10-31 12:45:48,503 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:45:48,503 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:45:48,507 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:45:49,699 INFO [INIT] Dropping privileges 2019-10-31 12:45:49,699 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:45:49,699 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:45:49,701 INFO [INIT] Marking all workers as offline 2019-10-31 12:45:49,733 INFO [INIT] LAVA master has started. 2019-10-31 12:45:49,733 INFO [INIT] Using protocol version 3 2019-10-31 12:45:57,951 DEBUG lava-logs => PING(20) 2019-10-31 12:46:05,329 DEBUG Dhanu => PING(20) 2019-10-31 12:46:05,330 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:46:09,738 INFO scheduling health checks: 2019-10-31 12:46:09,789 INFO scheduling jobs: 2019-10-31 12:46:09,791 DEBUG - qemu 2019-10-31 12:46:09,912 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:46:09,912 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:46:09,920 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:46:11,180 INFO [INIT] Dropping privileges 2019-10-31 12:46:11,180 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:46:11,180 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:46:11,182 INFO [INIT] Marking all workers as offline 2019-10-31 12:46:11,213 INFO [INIT] LAVA master has started. 2019-10-31 12:46:11,213 INFO [INIT] Using protocol version 3 2019-10-31 12:46:17,969 DEBUG lava-logs => PING(20) 2019-10-31 12:46:25,348 DEBUG Dhanu => PING(20) 2019-10-31 12:46:25,348 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:46:31,218 INFO scheduling health checks: 2019-10-31 12:46:31,258 INFO scheduling jobs: 2019-10-31 12:46:31,262 DEBUG - qemu 2019-10-31 12:46:31,359 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:46:31,359 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:46:31,365 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:46:32,835 INFO [INIT] Dropping privileges 2019-10-31 12:46:32,835 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:46:32,835 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:46:32,836 INFO [INIT] Marking all workers as offline 2019-10-31 12:46:32,872 INFO [INIT] LAVA master has started. 2019-10-31 12:46:32,874 INFO [INIT] Using protocol version 3 2019-10-31 12:46:37,984 DEBUG lava-logs => PING(20) 2019-10-31 12:46:45,368 DEBUG Dhanu => PING(20) 2019-10-31 12:46:45,368 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:46:52,882 INFO scheduling health checks: 2019-10-31 12:46:52,927 INFO scheduling jobs: 2019-10-31 12:46:52,930 DEBUG - qemu 2019-10-31 12:46:53,006 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:46:53,006 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:46:53,011 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:46:54,113 INFO [INIT] Dropping privileges 2019-10-31 12:46:54,113 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:46:54,113 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:46:54,115 INFO [INIT] Marking all workers as offline 2019-10-31 12:46:54,154 INFO [INIT] LAVA master has started. 2019-10-31 12:46:54,154 INFO [INIT] Using protocol version 3 2019-10-31 12:46:57,999 DEBUG lava-logs => PING(20) 2019-10-31 12:47:05,385 DEBUG Dhanu => PING(20) 2019-10-31 12:47:05,385 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:47:14,162 INFO scheduling health checks: 2019-10-31 12:47:14,196 INFO scheduling jobs: 2019-10-31 12:47:14,198 DEBUG - qemu 2019-10-31 12:47:14,296 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:47:14,296 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:47:14,304 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:47:15,719 INFO [INIT] Dropping privileges 2019-10-31 12:47:15,719 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:47:15,719 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:47:15,720 INFO [INIT] Marking all workers as offline 2019-10-31 12:47:15,753 INFO [INIT] LAVA master has started. 2019-10-31 12:47:15,754 INFO [INIT] Using protocol version 3 2019-10-31 12:47:18,012 DEBUG lava-logs => PING(20) 2019-10-31 12:47:25,402 DEBUG Dhanu => PING(20) 2019-10-31 12:47:25,403 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:47:35,761 INFO scheduling health checks: 2019-10-31 12:47:35,798 INFO scheduling jobs: 2019-10-31 12:47:35,803 DEBUG - qemu 2019-10-31 12:47:35,879 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:47:35,880 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:47:35,884 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:47:37,180 INFO [INIT] Dropping privileges 2019-10-31 12:47:37,180 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:47:37,180 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:47:37,182 INFO [INIT] Marking all workers as offline 2019-10-31 12:47:37,257 INFO [INIT] LAVA master has started. 2019-10-31 12:47:37,257 INFO [INIT] Using protocol version 3 2019-10-31 12:47:38,023 DEBUG lava-logs => PING(20) 2019-10-31 12:47:45,422 DEBUG Dhanu => PING(20) 2019-10-31 12:47:45,422 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:47:57,269 INFO scheduling health checks: 2019-10-31 12:47:57,300 INFO scheduling jobs: 2019-10-31 12:47:57,303 DEBUG - qemu 2019-10-31 12:47:57,377 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:47:57,378 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:47:57,382 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:47:58,651 INFO [INIT] Dropping privileges 2019-10-31 12:47:58,651 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:47:58,651 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:47:58,652 INFO [INIT] Marking all workers as offline 2019-10-31 12:47:58,692 INFO [INIT] LAVA master has started. 2019-10-31 12:47:58,692 INFO [INIT] Using protocol version 3 2019-10-31 12:47:58,758 DEBUG lava-logs => PING(20) 2019-10-31 12:48:05,442 DEBUG Dhanu => PING(20) 2019-10-31 12:48:05,443 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:48:18,706 INFO scheduling health checks: 2019-10-31 12:48:18,747 INFO scheduling jobs: 2019-10-31 12:48:18,750 DEBUG - qemu 2019-10-31 12:48:18,823 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:48:18,823 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:48:18,831 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:48:20,110 INFO [INIT] Dropping privileges 2019-10-31 12:48:20,110 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:48:20,110 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:48:20,112 INFO [INIT] Marking all workers as offline 2019-10-31 12:48:20,139 INFO [INIT] LAVA master has started. 2019-10-31 12:48:20,139 INFO [INIT] Using protocol version 3 2019-10-31 12:48:25,458 DEBUG Dhanu => PING(20) 2019-10-31 12:48:25,459 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:48:38,797 DEBUG lava-logs => PING(20) 2019-10-31 12:48:40,140 INFO scheduling health checks: 2019-10-31 12:48:40,194 INFO scheduling jobs: 2019-10-31 12:48:40,197 DEBUG - qemu 2019-10-31 12:48:40,275 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:48:40,275 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:48:40,280 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:48:41,340 INFO [INIT] Dropping privileges 2019-10-31 12:48:41,340 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:48:41,340 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:48:41,342 INFO [INIT] Marking all workers as offline 2019-10-31 12:48:41,385 INFO [INIT] LAVA master has started. 2019-10-31 12:48:41,385 INFO [INIT] Using protocol version 3 2019-10-31 12:48:45,477 DEBUG Dhanu => PING(20) 2019-10-31 12:48:45,477 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:48:58,816 DEBUG lava-logs => PING(20) 2019-10-31 12:49:01,388 INFO scheduling health checks: 2019-10-31 12:49:01,429 INFO scheduling jobs: 2019-10-31 12:49:01,432 DEBUG - qemu 2019-10-31 12:49:01,505 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:49:01,506 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:49:01,514 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:49:02,712 INFO [INIT] Dropping privileges 2019-10-31 12:49:02,712 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:49:02,712 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:49:02,714 INFO [INIT] Marking all workers as offline 2019-10-31 12:49:02,741 INFO [INIT] LAVA master has started. 2019-10-31 12:49:02,742 INFO [INIT] Using protocol version 3 2019-10-31 12:49:05,498 DEBUG Dhanu => PING(20) 2019-10-31 12:49:05,499 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:49:18,835 DEBUG lava-logs => PING(20) 2019-10-31 12:49:22,746 INFO scheduling health checks: 2019-10-31 12:49:22,775 INFO scheduling jobs: 2019-10-31 12:49:22,777 DEBUG - qemu 2019-10-31 12:49:22,850 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:49:22,850 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:49:22,858 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:49:24,109 INFO [INIT] Dropping privileges 2019-10-31 12:49:24,109 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:49:24,109 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:49:24,111 INFO [INIT] Marking all workers as offline 2019-10-31 12:49:24,155 INFO [INIT] LAVA master has started. 2019-10-31 12:49:24,156 INFO [INIT] Using protocol version 3 2019-10-31 12:49:25,514 DEBUG Dhanu => PING(20) 2019-10-31 12:49:25,514 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:49:38,850 DEBUG lava-logs => PING(20) 2019-10-31 12:49:44,161 INFO scheduling health checks: 2019-10-31 12:49:44,200 INFO scheduling jobs: 2019-10-31 12:49:44,203 DEBUG - qemu 2019-10-31 12:49:44,276 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:49:44,277 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:49:44,281 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:49:45,643 INFO [INIT] Dropping privileges 2019-10-31 12:49:45,643 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:49:45,643 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:49:45,645 INFO [INIT] Marking all workers as offline 2019-10-31 12:49:45,669 INFO [INIT] LAVA master has started. 2019-10-31 12:49:45,669 INFO [INIT] Using protocol version 3 2019-10-31 12:49:45,722 DEBUG Dhanu => PING(20) 2019-10-31 12:49:45,723 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:49:58,861 DEBUG lava-logs => PING(20) 2019-10-31 12:50:05,676 INFO scheduling health checks: 2019-10-31 12:50:05,702 INFO scheduling jobs: 2019-10-31 12:50:05,704 DEBUG - qemu 2019-10-31 12:50:05,775 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:50:05,775 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:50:05,780 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:50:07,096 INFO [INIT] Dropping privileges 2019-10-31 12:50:07,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:50:07,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:50:07,099 INFO [INIT] Marking all workers as offline 2019-10-31 12:50:07,136 INFO [INIT] LAVA master has started. 2019-10-31 12:50:07,136 INFO [INIT] Using protocol version 3 2019-10-31 12:50:18,874 DEBUG lava-logs => PING(20) 2019-10-31 12:50:25,758 DEBUG Dhanu => PING(20) 2019-10-31 12:50:25,760 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:50:27,137 INFO scheduling health checks: 2019-10-31 12:50:27,193 INFO scheduling jobs: 2019-10-31 12:50:27,194 DEBUG - qemu 2019-10-31 12:50:27,264 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:50:27,264 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:50:27,273 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:50:28,424 INFO [INIT] Dropping privileges 2019-10-31 12:50:28,425 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:50:28,425 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:50:28,427 INFO [INIT] Marking all workers as offline 2019-10-31 12:50:28,474 INFO [INIT] LAVA master has started. 2019-10-31 12:50:28,474 INFO [INIT] Using protocol version 3 2019-10-31 12:50:38,887 DEBUG lava-logs => PING(20) 2019-10-31 12:50:45,780 DEBUG Dhanu => PING(20) 2019-10-31 12:50:45,780 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:50:48,477 INFO scheduling health checks: 2019-10-31 12:50:48,517 INFO scheduling jobs: 2019-10-31 12:50:48,519 DEBUG - qemu 2019-10-31 12:50:48,607 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:50:48,608 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:50:48,613 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:50:49,943 INFO [INIT] Dropping privileges 2019-10-31 12:50:49,944 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:50:49,944 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:50:49,946 INFO [INIT] Marking all workers as offline 2019-10-31 12:50:49,975 INFO [INIT] LAVA master has started. 2019-10-31 12:50:49,976 INFO [INIT] Using protocol version 3 2019-10-31 12:50:58,906 DEBUG lava-logs => PING(20) 2019-10-31 12:51:05,798 DEBUG Dhanu => PING(20) 2019-10-31 12:51:05,800 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:51:09,981 INFO scheduling health checks: 2019-10-31 12:51:10,019 INFO scheduling jobs: 2019-10-31 12:51:10,022 DEBUG - qemu 2019-10-31 12:51:10,096 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:51:10,097 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:51:10,101 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:51:11,369 INFO [INIT] Dropping privileges 2019-10-31 12:51:11,370 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:51:11,370 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:51:11,371 INFO [INIT] Marking all workers as offline 2019-10-31 12:51:11,411 INFO [INIT] LAVA master has started. 2019-10-31 12:51:11,412 INFO [INIT] Using protocol version 3 2019-10-31 12:51:18,918 DEBUG lava-logs => PING(20) 2019-10-31 12:51:25,818 DEBUG Dhanu => PING(20) 2019-10-31 12:51:25,818 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:51:31,417 INFO scheduling health checks: 2019-10-31 12:51:31,477 INFO scheduling jobs: 2019-10-31 12:51:31,479 DEBUG - qemu 2019-10-31 12:51:31,558 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:51:31,558 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:51:31,562 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:51:32,915 INFO [INIT] Dropping privileges 2019-10-31 12:51:32,915 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:51:32,915 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:51:32,917 INFO [INIT] Marking all workers as offline 2019-10-31 12:51:32,945 INFO [INIT] LAVA master has started. 2019-10-31 12:51:32,946 INFO [INIT] Using protocol version 3 2019-10-31 12:51:38,937 DEBUG lava-logs => PING(20) 2019-10-31 12:51:45,833 DEBUG Dhanu => PING(20) 2019-10-31 12:51:45,833 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:51:52,953 INFO scheduling health checks: 2019-10-31 12:51:52,990 INFO scheduling jobs: 2019-10-31 12:51:52,993 DEBUG - qemu 2019-10-31 12:51:53,065 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:51:53,065 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:51:53,070 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:51:54,349 INFO [INIT] Dropping privileges 2019-10-31 12:51:54,350 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:51:54,350 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:51:54,351 INFO [INIT] Marking all workers as offline 2019-10-31 12:51:54,437 INFO [INIT] LAVA master has started. 2019-10-31 12:51:54,438 INFO [INIT] Using protocol version 3 2019-10-31 12:51:58,954 DEBUG lava-logs => PING(20) 2019-10-31 12:52:05,851 DEBUG Dhanu => PING(20) 2019-10-31 12:52:05,852 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:52:14,447 INFO scheduling health checks: 2019-10-31 12:52:14,481 INFO scheduling jobs: 2019-10-31 12:52:14,483 DEBUG - qemu 2019-10-31 12:52:14,555 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:52:14,555 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:52:14,559 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:52:15,750 INFO [INIT] Dropping privileges 2019-10-31 12:52:15,750 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:52:15,750 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:52:15,752 INFO [INIT] Marking all workers as offline 2019-10-31 12:52:15,793 INFO [INIT] LAVA master has started. 2019-10-31 12:52:15,794 INFO [INIT] Using protocol version 3 2019-10-31 12:52:18,971 DEBUG lava-logs => PING(20) 2019-10-31 12:52:25,871 DEBUG Dhanu => PING(20) 2019-10-31 12:52:25,872 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:52:35,803 INFO scheduling health checks: 2019-10-31 12:52:35,839 INFO scheduling jobs: 2019-10-31 12:52:35,842 DEBUG - qemu 2019-10-31 12:52:35,914 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:52:35,915 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:52:35,919 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:52:37,095 INFO [INIT] Dropping privileges 2019-10-31 12:52:37,095 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:52:37,095 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:52:37,097 INFO [INIT] Marking all workers as offline 2019-10-31 12:52:37,130 INFO [INIT] LAVA master has started. 2019-10-31 12:52:37,130 INFO [INIT] Using protocol version 3 2019-10-31 12:52:38,985 DEBUG lava-logs => PING(20) 2019-10-31 12:52:45,894 DEBUG Dhanu => PING(20) 2019-10-31 12:52:45,895 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:52:57,141 INFO scheduling health checks: 2019-10-31 12:52:57,174 INFO scheduling jobs: 2019-10-31 12:52:57,177 DEBUG - qemu 2019-10-31 12:52:57,250 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:52:57,250 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:52:57,254 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:52:58,341 INFO [INIT] Dropping privileges 2019-10-31 12:52:58,341 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:52:58,341 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:52:58,343 INFO [INIT] Marking all workers as offline 2019-10-31 12:52:58,376 INFO [INIT] LAVA master has started. 2019-10-31 12:52:58,376 INFO [INIT] Using protocol version 3 2019-10-31 12:52:59,000 DEBUG lava-logs => PING(20) 2019-10-31 12:53:05,913 DEBUG Dhanu => PING(20) 2019-10-31 12:53:05,913 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:53:18,389 INFO scheduling health checks: 2019-10-31 12:53:18,421 INFO scheduling jobs: 2019-10-31 12:53:18,423 DEBUG - qemu 2019-10-31 12:53:18,496 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:53:18,496 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:53:18,501 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:53:19,587 INFO [INIT] Dropping privileges 2019-10-31 12:53:19,587 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:53:19,587 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:53:19,589 INFO [INIT] Marking all workers as offline 2019-10-31 12:53:19,634 INFO [INIT] LAVA master has started. 2019-10-31 12:53:19,634 INFO [INIT] Using protocol version 3 2019-10-31 12:53:19,688 DEBUG lava-logs => PING(20) 2019-10-31 12:53:25,931 DEBUG Dhanu => PING(20) 2019-10-31 12:53:25,932 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:53:39,648 INFO scheduling health checks: 2019-10-31 12:53:39,678 INFO scheduling jobs: 2019-10-31 12:53:39,680 DEBUG - qemu 2019-10-31 12:53:39,757 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:53:39,757 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:53:39,761 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:53:40,834 INFO [INIT] Dropping privileges 2019-10-31 12:53:40,834 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:53:40,834 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:53:40,836 INFO [INIT] Marking all workers as offline 2019-10-31 12:53:40,904 INFO [INIT] LAVA master has started. 2019-10-31 12:53:40,904 INFO [INIT] Using protocol version 3 2019-10-31 12:53:45,953 DEBUG Dhanu => PING(20) 2019-10-31 12:53:45,954 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:53:59,731 DEBUG lava-logs => PING(20) 2019-10-31 12:54:00,905 INFO scheduling health checks: 2019-10-31 12:54:00,925 INFO scheduling jobs: 2019-10-31 12:54:00,926 DEBUG - qemu 2019-10-31 12:54:00,991 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:54:00,991 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:54:01,001 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:54:02,095 INFO [INIT] Dropping privileges 2019-10-31 12:54:02,096 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:54:02,096 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:54:02,098 INFO [INIT] Marking all workers as offline 2019-10-31 12:54:02,162 INFO [INIT] LAVA master has started. 2019-10-31 12:54:02,163 INFO [INIT] Using protocol version 3 2019-10-31 12:54:05,974 DEBUG Dhanu => PING(20) 2019-10-31 12:54:05,974 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:54:19,748 DEBUG lava-logs => PING(20) 2019-10-31 12:54:22,166 INFO scheduling health checks: 2019-10-31 12:54:22,194 INFO scheduling jobs: 2019-10-31 12:54:22,195 DEBUG - qemu 2019-10-31 12:54:22,262 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:54:22,262 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:54:22,277 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:54:23,345 INFO [INIT] Dropping privileges 2019-10-31 12:54:23,345 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:54:23,346 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:54:23,347 INFO [INIT] Marking all workers as offline 2019-10-31 12:54:23,384 INFO [INIT] LAVA master has started. 2019-10-31 12:54:23,385 INFO [INIT] Using protocol version 3 2019-10-31 12:54:25,994 DEBUG Dhanu => PING(20) 2019-10-31 12:54:25,995 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:54:39,766 DEBUG lava-logs => PING(20) 2019-10-31 12:54:43,388 INFO scheduling health checks: 2019-10-31 12:54:43,418 INFO scheduling jobs: 2019-10-31 12:54:43,420 DEBUG - qemu 2019-10-31 12:54:43,493 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:54:43,493 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:54:43,502 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:54:44,588 INFO [INIT] Dropping privileges 2019-10-31 12:54:44,588 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:54:44,588 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:54:44,590 INFO [INIT] Marking all workers as offline 2019-10-31 12:54:44,620 INFO [INIT] LAVA master has started. 2019-10-31 12:54:44,621 INFO [INIT] Using protocol version 3 2019-10-31 12:54:46,015 DEBUG Dhanu => PING(20) 2019-10-31 12:54:46,015 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:54:59,783 DEBUG lava-logs => PING(20) 2019-10-31 12:55:04,626 INFO scheduling health checks: 2019-10-31 12:55:04,654 INFO scheduling jobs: 2019-10-31 12:55:04,656 DEBUG - qemu 2019-10-31 12:55:04,729 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:55:04,729 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:55:04,734 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:55:05,850 INFO [INIT] Dropping privileges 2019-10-31 12:55:05,850 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:55:05,850 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:55:05,852 INFO [INIT] Marking all workers as offline 2019-10-31 12:55:05,888 INFO [INIT] LAVA master has started. 2019-10-31 12:55:05,888 INFO [INIT] Using protocol version 3 2019-10-31 12:55:06,031 DEBUG Dhanu => PING(20) 2019-10-31 12:55:06,031 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:55:19,800 DEBUG lava-logs => PING(20) 2019-10-31 12:55:25,895 INFO scheduling health checks: 2019-10-31 12:55:25,934 INFO scheduling jobs: 2019-10-31 12:55:25,936 DEBUG - qemu 2019-10-31 12:55:26,008 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:55:26,008 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:55:26,013 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:55:27,093 INFO [INIT] Dropping privileges 2019-10-31 12:55:27,093 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:55:27,093 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:55:27,095 INFO [INIT] Marking all workers as offline 2019-10-31 12:55:27,124 INFO [INIT] LAVA master has started. 2019-10-31 12:55:27,124 INFO [INIT] Using protocol version 3 2019-10-31 12:55:27,208 DEBUG Dhanu => PING(20) 2019-10-31 12:55:27,209 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:55:39,816 DEBUG lava-logs => PING(20) 2019-10-31 12:55:47,129 INFO scheduling health checks: 2019-10-31 12:55:47,169 INFO scheduling jobs: 2019-10-31 12:55:47,171 DEBUG - qemu 2019-10-31 12:55:47,238 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:55:47,238 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:55:47,243 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:55:48,350 INFO [INIT] Dropping privileges 2019-10-31 12:55:48,350 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:55:48,350 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:55:48,352 INFO [INIT] Marking all workers as offline 2019-10-31 12:55:48,382 INFO [INIT] LAVA master has started. 2019-10-31 12:55:48,382 INFO [INIT] Using protocol version 3 2019-10-31 12:55:59,830 DEBUG lava-logs => PING(20) 2019-10-31 12:56:07,251 DEBUG Dhanu => PING(20) 2019-10-31 12:56:07,252 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:56:08,383 INFO scheduling health checks: 2019-10-31 12:56:08,404 INFO scheduling jobs: 2019-10-31 12:56:08,406 DEBUG - qemu 2019-10-31 12:56:08,475 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:56:08,476 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:56:08,480 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:56:09,596 INFO [INIT] Dropping privileges 2019-10-31 12:56:09,596 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:56:09,596 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:56:09,597 INFO [INIT] Marking all workers as offline 2019-10-31 12:56:09,640 INFO [INIT] LAVA master has started. 2019-10-31 12:56:09,641 INFO [INIT] Using protocol version 3 2019-10-31 12:56:19,843 DEBUG lava-logs => PING(20) 2019-10-31 12:56:27,273 DEBUG Dhanu => PING(20) 2019-10-31 12:56:27,274 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:56:29,643 INFO scheduling health checks: 2019-10-31 12:56:29,662 INFO scheduling jobs: 2019-10-31 12:56:29,663 DEBUG - qemu 2019-10-31 12:56:29,734 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:56:29,734 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:56:29,739 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:56:30,976 INFO [INIT] Dropping privileges 2019-10-31 12:56:30,976 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:56:30,976 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:56:30,978 INFO [INIT] Marking all workers as offline 2019-10-31 12:56:31,010 INFO [INIT] LAVA master has started. 2019-10-31 12:56:31,010 INFO [INIT] Using protocol version 3 2019-10-31 12:56:39,864 DEBUG lava-logs => PING(20) 2019-10-31 12:56:47,294 DEBUG Dhanu => PING(20) 2019-10-31 12:56:47,294 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:56:51,013 INFO scheduling health checks: 2019-10-31 12:56:51,042 INFO scheduling jobs: 2019-10-31 12:56:51,044 DEBUG - qemu 2019-10-31 12:56:51,118 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:56:51,118 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:56:51,126 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:56:52,334 INFO [INIT] Dropping privileges 2019-10-31 12:56:52,335 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:56:52,335 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:56:52,337 INFO [INIT] Marking all workers as offline 2019-10-31 12:56:52,389 INFO [INIT] LAVA master has started. 2019-10-31 12:56:52,390 INFO [INIT] Using protocol version 3 2019-10-31 12:56:59,885 DEBUG lava-logs => PING(20) 2019-10-31 12:57:07,315 DEBUG Dhanu => PING(20) 2019-10-31 12:57:07,316 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:57:12,395 INFO scheduling health checks: 2019-10-31 12:57:12,433 INFO scheduling jobs: 2019-10-31 12:57:12,435 DEBUG - qemu 2019-10-31 12:57:12,510 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:57:12,510 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:57:12,518 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:57:13,588 INFO [INIT] Dropping privileges 2019-10-31 12:57:13,588 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:57:13,588 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:57:13,590 INFO [INIT] Marking all workers as offline 2019-10-31 12:57:13,635 INFO [INIT] LAVA master has started. 2019-10-31 12:57:13,636 INFO [INIT] Using protocol version 3 2019-10-31 12:57:19,902 DEBUG lava-logs => PING(20) 2019-10-31 12:57:27,339 DEBUG Dhanu => PING(20) 2019-10-31 12:57:27,339 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:57:33,642 INFO scheduling health checks: 2019-10-31 12:57:33,680 INFO scheduling jobs: 2019-10-31 12:57:33,683 DEBUG - qemu 2019-10-31 12:57:33,757 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:57:33,757 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:57:33,766 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:57:34,850 INFO [INIT] Dropping privileges 2019-10-31 12:57:34,850 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:57:34,850 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:57:34,852 INFO [INIT] Marking all workers as offline 2019-10-31 12:57:34,881 INFO [INIT] LAVA master has started. 2019-10-31 12:57:34,881 INFO [INIT] Using protocol version 3 2019-10-31 12:57:39,918 DEBUG lava-logs => PING(20) 2019-10-31 12:57:47,360 DEBUG Dhanu => PING(20) 2019-10-31 12:57:47,361 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:57:54,889 INFO scheduling health checks: 2019-10-31 12:57:54,926 INFO scheduling jobs: 2019-10-31 12:57:54,928 DEBUG - qemu 2019-10-31 12:57:54,994 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:57:54,994 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:57:54,999 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:57:56,080 INFO [INIT] Dropping privileges 2019-10-31 12:57:56,080 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:57:56,080 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:57:56,082 INFO [INIT] Marking all workers as offline 2019-10-31 12:57:56,106 INFO [INIT] LAVA master has started. 2019-10-31 12:57:56,106 INFO [INIT] Using protocol version 3 2019-10-31 12:57:59,934 DEBUG lava-logs => PING(20) 2019-10-31 12:58:07,381 DEBUG Dhanu => PING(20) 2019-10-31 12:58:07,381 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:58:16,115 INFO scheduling health checks: 2019-10-31 12:58:16,139 INFO scheduling jobs: 2019-10-31 12:58:16,142 DEBUG - qemu 2019-10-31 12:58:16,215 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:58:16,215 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:58:16,219 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:58:17,343 INFO [INIT] Dropping privileges 2019-10-31 12:58:17,343 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:58:17,343 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:58:17,344 INFO [INIT] Marking all workers as offline 2019-10-31 12:58:17,375 INFO [INIT] LAVA master has started. 2019-10-31 12:58:17,376 INFO [INIT] Using protocol version 3 2019-10-31 12:58:19,950 DEBUG lava-logs => PING(20) 2019-10-31 12:58:27,397 DEBUG Dhanu => PING(20) 2019-10-31 12:58:27,397 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:58:37,386 INFO scheduling health checks: 2019-10-31 12:58:37,408 INFO scheduling jobs: 2019-10-31 12:58:37,410 DEBUG - qemu 2019-10-31 12:58:37,479 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:58:37,479 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:58:37,484 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:58:38,586 INFO [INIT] Dropping privileges 2019-10-31 12:58:38,587 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:58:38,587 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:58:38,589 INFO [INIT] Marking all workers as offline 2019-10-31 12:58:38,645 INFO [INIT] LAVA master has started. 2019-10-31 12:58:38,646 INFO [INIT] Using protocol version 3 2019-10-31 12:58:39,956 DEBUG lava-logs => PING(20) 2019-10-31 12:58:47,416 DEBUG Dhanu => PING(20) 2019-10-31 12:58:47,416 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:58:58,657 INFO scheduling health checks: 2019-10-31 12:58:58,688 INFO scheduling jobs: 2019-10-31 12:58:58,690 DEBUG - qemu 2019-10-31 12:58:58,760 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:58:58,760 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:58:58,764 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:58:59,846 INFO [INIT] Dropping privileges 2019-10-31 12:58:59,846 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:58:59,846 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:58:59,848 INFO [INIT] Marking all workers as offline 2019-10-31 12:58:59,891 INFO [INIT] LAVA master has started. 2019-10-31 12:58:59,891 INFO [INIT] Using protocol version 3 2019-10-31 12:58:59,970 DEBUG lava-logs => PING(20) 2019-10-31 12:59:07,434 DEBUG Dhanu => PING(20) 2019-10-31 12:59:07,435 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:59:19,901 INFO scheduling health checks: 2019-10-31 12:59:19,934 INFO scheduling jobs: 2019-10-31 12:59:19,936 DEBUG - qemu 2019-10-31 12:59:20,010 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:59:20,010 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:59:20,015 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:59:21,089 INFO [INIT] Dropping privileges 2019-10-31 12:59:21,089 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:59:21,089 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:59:21,091 INFO [INIT] Marking all workers as offline 2019-10-31 12:59:21,115 INFO [INIT] LAVA master has started. 2019-10-31 12:59:21,115 INFO [INIT] Using protocol version 3 2019-10-31 12:59:27,446 DEBUG Dhanu => PING(20) 2019-10-31 12:59:27,447 WARNING Unknown dispatcher (server crashed) 2019-10-31 12:59:40,010 DEBUG lava-logs => PING(20) 2019-10-31 12:59:41,117 INFO scheduling health checks: 2019-10-31 12:59:41,136 INFO scheduling jobs: 2019-10-31 12:59:41,138 DEBUG - qemu 2019-10-31 12:59:41,207 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 12:59:41,207 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 12:59:41,211 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:59:42,365 INFO [INIT] Dropping privileges 2019-10-31 12:59:42,365 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:59:42,365 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:59:42,366 INFO [INIT] Marking all workers as offline 2019-10-31 12:59:42,395 INFO [INIT] LAVA master has started. 2019-10-31 12:59:42,395 INFO [INIT] Using protocol version 3 2019-10-31 12:59:47,460 DEBUG Dhanu => PING(20) 2019-10-31 12:59:47,460 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:00:00,031 DEBUG lava-logs => PING(20) 2019-10-31 13:00:02,398 INFO scheduling health checks: 2019-10-31 13:00:02,428 INFO scheduling jobs: 2019-10-31 13:00:02,430 DEBUG - qemu 2019-10-31 13:00:02,503 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:00:02,503 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:00:02,508 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:00:03,582 INFO [INIT] Dropping privileges 2019-10-31 13:00:03,583 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:00:03,583 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:00:03,584 INFO [INIT] Marking all workers as offline 2019-10-31 13:00:03,630 INFO [INIT] LAVA master has started. 2019-10-31 13:00:03,630 INFO [INIT] Using protocol version 3 2019-10-31 13:00:07,482 DEBUG Dhanu => PING(20) 2019-10-31 13:00:07,483 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:00:20,049 DEBUG lava-logs => PING(20) 2019-10-31 13:00:23,634 INFO scheduling health checks: 2019-10-31 13:00:23,686 INFO scheduling jobs: 2019-10-31 13:00:23,689 DEBUG - qemu 2019-10-31 13:00:23,762 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:00:23,762 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:00:23,767 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:00:24,840 INFO [INIT] Dropping privileges 2019-10-31 13:00:24,840 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:00:24,840 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:00:24,842 INFO [INIT] Marking all workers as offline 2019-10-31 13:00:24,888 INFO [INIT] LAVA master has started. 2019-10-31 13:00:24,888 INFO [INIT] Using protocol version 3 2019-10-31 13:00:27,504 DEBUG Dhanu => PING(20) 2019-10-31 13:00:27,504 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:00:40,067 DEBUG lava-logs => PING(20) 2019-10-31 13:00:44,893 INFO scheduling health checks: 2019-10-31 13:00:44,921 INFO scheduling jobs: 2019-10-31 13:00:44,922 DEBUG - qemu 2019-10-31 13:00:44,994 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:00:44,994 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:00:44,999 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:00:46,084 INFO [INIT] Dropping privileges 2019-10-31 13:00:46,085 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:00:46,085 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:00:46,086 INFO [INIT] Marking all workers as offline 2019-10-31 13:00:46,134 INFO [INIT] LAVA master has started. 2019-10-31 13:00:46,135 INFO [INIT] Using protocol version 3 2019-10-31 13:00:47,523 DEBUG Dhanu => PING(20) 2019-10-31 13:00:47,524 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:01:00,084 DEBUG lava-logs => PING(20) 2019-10-31 13:01:06,141 INFO scheduling health checks: 2019-10-31 13:01:06,178 INFO scheduling jobs: 2019-10-31 13:01:06,182 DEBUG - qemu 2019-10-31 13:01:06,255 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:01:06,255 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:01:06,259 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:01:07,349 INFO [INIT] Dropping privileges 2019-10-31 13:01:07,349 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:01:07,349 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:01:07,351 INFO [INIT] Marking all workers as offline 2019-10-31 13:01:07,380 INFO [INIT] LAVA master has started. 2019-10-31 13:01:07,380 INFO [INIT] Using protocol version 3 2019-10-31 13:01:07,542 DEBUG Dhanu => PING(20) 2019-10-31 13:01:07,543 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:01:20,100 DEBUG lava-logs => PING(20) 2019-10-31 13:01:27,388 INFO scheduling health checks: 2019-10-31 13:01:27,425 INFO scheduling jobs: 2019-10-31 13:01:27,428 DEBUG - qemu 2019-10-31 13:01:27,503 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:01:27,503 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:01:27,508 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:01:28,606 INFO [INIT] Dropping privileges 2019-10-31 13:01:28,606 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:01:28,606 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:01:28,608 INFO [INIT] Marking all workers as offline 2019-10-31 13:01:28,638 INFO [INIT] LAVA master has started. 2019-10-31 13:01:28,638 INFO [INIT] Using protocol version 3 2019-10-31 13:01:28,716 DEBUG Dhanu => PING(20) 2019-10-31 13:01:28,717 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:01:40,114 DEBUG lava-logs => PING(20) 2019-10-31 13:01:48,645 INFO scheduling health checks: 2019-10-31 13:01:48,671 INFO scheduling jobs: 2019-10-31 13:01:48,673 DEBUG - qemu 2019-10-31 13:01:48,737 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:01:48,738 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:01:48,742 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:01:49,840 INFO [INIT] Dropping privileges 2019-10-31 13:01:49,840 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:01:49,840 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:01:49,842 INFO [INIT] Marking all workers as offline 2019-10-31 13:01:49,874 INFO [INIT] LAVA master has started. 2019-10-31 13:01:49,874 INFO [INIT] Using protocol version 3 2019-10-31 13:02:00,127 DEBUG lava-logs => PING(20) 2019-10-31 13:02:08,762 DEBUG Dhanu => PING(20) 2019-10-31 13:02:08,762 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:02:09,875 INFO scheduling health checks: 2019-10-31 13:02:09,907 INFO scheduling jobs: 2019-10-31 13:02:09,909 DEBUG - qemu 2019-10-31 13:02:09,984 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:02:09,984 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:02:09,988 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:02:11,100 INFO [INIT] Dropping privileges 2019-10-31 13:02:11,100 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:02:11,100 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:02:11,102 INFO [INIT] Marking all workers as offline 2019-10-31 13:02:11,132 INFO [INIT] LAVA master has started. 2019-10-31 13:02:11,132 INFO [INIT] Using protocol version 3 2019-10-31 13:02:20,148 DEBUG lava-logs => PING(20) 2019-10-31 13:02:28,783 DEBUG Dhanu => PING(20) 2019-10-31 13:02:28,784 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:02:31,135 INFO scheduling health checks: 2019-10-31 13:02:31,176 INFO scheduling jobs: 2019-10-31 13:02:31,179 DEBUG - qemu 2019-10-31 13:02:31,249 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:02:31,249 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:02:31,259 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:02:32,342 INFO [INIT] Dropping privileges 2019-10-31 13:02:32,342 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:02:32,342 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:02:32,344 INFO [INIT] Marking all workers as offline 2019-10-31 13:02:32,377 INFO [INIT] LAVA master has started. 2019-10-31 13:02:32,378 INFO [INIT] Using protocol version 3 2019-10-31 13:02:40,169 DEBUG lava-logs => PING(20) 2019-10-31 13:02:48,802 DEBUG Dhanu => PING(20) 2019-10-31 13:02:48,803 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:02:52,381 INFO scheduling health checks: 2019-10-31 13:02:52,433 INFO scheduling jobs: 2019-10-31 13:02:52,435 DEBUG - qemu 2019-10-31 13:02:52,509 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:02:52,510 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:02:52,517 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:02:53,590 INFO [INIT] Dropping privileges 2019-10-31 13:02:53,590 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:02:53,590 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:02:53,592 INFO [INIT] Marking all workers as offline 2019-10-31 13:02:53,625 INFO [INIT] LAVA master has started. 2019-10-31 13:02:53,625 INFO [INIT] Using protocol version 3 2019-10-31 13:03:00,188 DEBUG lava-logs => PING(20) 2019-10-31 13:03:08,822 DEBUG Dhanu => PING(20) 2019-10-31 13:03:08,822 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:03:13,631 INFO scheduling health checks: 2019-10-31 13:03:13,658 INFO scheduling jobs: 2019-10-31 13:03:13,660 DEBUG - qemu 2019-10-31 13:03:13,732 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:03:13,733 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:03:13,744 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:03:14,863 INFO [INIT] Dropping privileges 2019-10-31 13:03:14,863 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:03:14,863 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:03:14,865 INFO [INIT] Marking all workers as offline 2019-10-31 13:03:14,893 INFO [INIT] LAVA master has started. 2019-10-31 13:03:14,894 INFO [INIT] Using protocol version 3 2019-10-31 13:03:20,204 DEBUG lava-logs => PING(20) 2019-10-31 13:03:28,844 DEBUG Dhanu => PING(20) 2019-10-31 13:03:28,845 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:03:34,900 INFO scheduling health checks: 2019-10-31 13:03:34,938 INFO scheduling jobs: 2019-10-31 13:03:34,941 DEBUG - qemu 2019-10-31 13:03:35,014 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:03:35,014 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:03:35,018 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:03:36,097 INFO [INIT] Dropping privileges 2019-10-31 13:03:36,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:03:36,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:03:36,099 INFO [INIT] Marking all workers as offline 2019-10-31 13:03:36,163 INFO [INIT] LAVA master has started. 2019-10-31 13:03:36,163 INFO [INIT] Using protocol version 3 2019-10-31 13:03:40,221 DEBUG lava-logs => PING(20) 2019-10-31 13:03:48,861 DEBUG Dhanu => PING(20) 2019-10-31 13:03:48,862 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:03:56,169 INFO scheduling health checks: 2019-10-31 13:03:56,206 INFO scheduling jobs: 2019-10-31 13:03:56,208 DEBUG - qemu 2019-10-31 13:03:56,276 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:03:56,276 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:03:56,281 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:03:57,342 INFO [INIT] Dropping privileges 2019-10-31 13:03:57,343 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:03:57,343 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:03:57,344 INFO [INIT] Marking all workers as offline 2019-10-31 13:03:57,387 INFO [INIT] LAVA master has started. 2019-10-31 13:03:57,387 INFO [INIT] Using protocol version 3 2019-10-31 13:04:00,234 DEBUG lava-logs => PING(20) 2019-10-31 13:04:08,879 DEBUG Dhanu => PING(20) 2019-10-31 13:04:08,880 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:04:17,396 INFO scheduling health checks: 2019-10-31 13:04:17,431 INFO scheduling jobs: 2019-10-31 13:04:17,433 DEBUG - qemu 2019-10-31 13:04:17,507 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:04:17,507 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:04:17,511 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:04:18,600 INFO [INIT] Dropping privileges 2019-10-31 13:04:18,600 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:04:18,600 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:04:18,602 INFO [INIT] Marking all workers as offline 2019-10-31 13:04:18,667 INFO [INIT] LAVA master has started. 2019-10-31 13:04:18,667 INFO [INIT] Using protocol version 3 2019-10-31 13:04:20,249 DEBUG lava-logs => PING(20) 2019-10-31 13:04:28,899 DEBUG Dhanu => PING(20) 2019-10-31 13:04:28,899 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:04:38,677 INFO scheduling health checks: 2019-10-31 13:04:38,711 INFO scheduling jobs: 2019-10-31 13:04:38,713 DEBUG - qemu 2019-10-31 13:04:38,787 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:04:38,788 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:04:38,792 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:04:40,090 INFO [INIT] Dropping privileges 2019-10-31 13:04:40,090 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:04:40,090 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:04:40,092 INFO [INIT] Marking all workers as offline 2019-10-31 13:04:40,124 INFO [INIT] LAVA master has started. 2019-10-31 13:04:40,125 INFO [INIT] Using protocol version 3 2019-10-31 13:04:40,262 DEBUG lava-logs => PING(20) 2019-10-31 13:04:48,920 DEBUG Dhanu => PING(20) 2019-10-31 13:04:48,920 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:05:00,136 INFO scheduling health checks: 2019-10-31 13:05:00,168 INFO scheduling jobs: 2019-10-31 13:05:00,170 DEBUG - qemu 2019-10-31 13:05:00,242 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:05:00,242 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:05:00,252 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:05:01,351 INFO [INIT] Dropping privileges 2019-10-31 13:05:01,351 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:05:01,351 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:05:01,353 INFO [INIT] Marking all workers as offline 2019-10-31 13:05:01,381 INFO [INIT] LAVA master has started. 2019-10-31 13:05:01,381 INFO [INIT] Using protocol version 3 2019-10-31 13:05:01,467 DEBUG lava-logs => PING(20) 2019-10-31 13:05:08,941 DEBUG Dhanu => PING(20) 2019-10-31 13:05:08,942 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:05:21,394 INFO scheduling health checks: 2019-10-31 13:05:21,426 INFO scheduling jobs: 2019-10-31 13:05:21,428 DEBUG - qemu 2019-10-31 13:05:21,502 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:05:21,502 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:05:21,507 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:05:22,593 INFO [INIT] Dropping privileges 2019-10-31 13:05:22,593 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:05:22,593 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:05:22,595 INFO [INIT] Marking all workers as offline 2019-10-31 13:05:22,629 INFO [INIT] LAVA master has started. 2019-10-31 13:05:22,629 INFO [INIT] Using protocol version 3 2019-10-31 13:05:28,961 DEBUG Dhanu => PING(20) 2019-10-31 13:05:28,962 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:05:41,510 DEBUG lava-logs => PING(20) 2019-10-31 13:05:42,631 INFO scheduling health checks: 2019-10-31 13:05:42,650 INFO scheduling jobs: 2019-10-31 13:05:42,651 DEBUG - qemu 2019-10-31 13:05:42,722 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:05:42,722 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:05:42,726 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:05:43,853 INFO [INIT] Dropping privileges 2019-10-31 13:05:43,853 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:05:43,853 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:05:43,855 INFO [INIT] Marking all workers as offline 2019-10-31 13:05:43,886 INFO [INIT] LAVA master has started. 2019-10-31 13:05:43,886 INFO [INIT] Using protocol version 3 2019-10-31 13:05:48,985 DEBUG Dhanu => PING(20) 2019-10-31 13:05:48,986 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:06:01,531 DEBUG lava-logs => PING(20) 2019-10-31 13:06:03,889 INFO scheduling health checks: 2019-10-31 13:06:03,952 INFO scheduling jobs: 2019-10-31 13:06:03,956 DEBUG - qemu 2019-10-31 13:06:04,030 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:06:04,031 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:06:04,035 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:06:05,113 INFO [INIT] Dropping privileges 2019-10-31 13:06:05,114 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:06:05,114 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:06:05,115 INFO [INIT] Marking all workers as offline 2019-10-31 13:06:05,143 INFO [INIT] LAVA master has started. 2019-10-31 13:06:05,143 INFO [INIT] Using protocol version 3 2019-10-31 13:06:09,002 DEBUG Dhanu => PING(20) 2019-10-31 13:06:09,002 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:06:21,551 DEBUG lava-logs => PING(20) 2019-10-31 13:06:25,147 INFO scheduling health checks: 2019-10-31 13:06:25,177 INFO scheduling jobs: 2019-10-31 13:06:25,178 DEBUG - qemu 2019-10-31 13:06:25,248 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:06:25,248 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:06:25,252 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:06:26,331 INFO [INIT] Dropping privileges 2019-10-31 13:06:26,331 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:06:26,331 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:06:26,332 INFO [INIT] Marking all workers as offline 2019-10-31 13:06:26,368 INFO [INIT] LAVA master has started. 2019-10-31 13:06:26,368 INFO [INIT] Using protocol version 3 2019-10-31 13:06:29,017 DEBUG Dhanu => PING(20) 2019-10-31 13:06:29,018 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:06:41,568 DEBUG lava-logs => PING(20) 2019-10-31 13:06:46,373 INFO scheduling health checks: 2019-10-31 13:06:46,401 INFO scheduling jobs: 2019-10-31 13:06:46,403 DEBUG - qemu 2019-10-31 13:06:46,475 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:06:46,476 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:06:46,480 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:06:47,588 INFO [INIT] Dropping privileges 2019-10-31 13:06:47,589 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:06:47,589 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:06:47,591 INFO [INIT] Marking all workers as offline 2019-10-31 13:06:47,626 INFO [INIT] LAVA master has started. 2019-10-31 13:06:47,626 INFO [INIT] Using protocol version 3 2019-10-31 13:06:49,035 DEBUG Dhanu => PING(20) 2019-10-31 13:06:49,036 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:07:01,585 DEBUG lava-logs => PING(20) 2019-10-31 13:07:07,632 INFO scheduling health checks: 2019-10-31 13:07:07,658 INFO scheduling jobs: 2019-10-31 13:07:07,660 DEBUG - qemu 2019-10-31 13:07:07,732 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:07:07,732 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:07:07,737 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:07:08,835 INFO [INIT] Dropping privileges 2019-10-31 13:07:08,835 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:07:08,835 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:07:08,837 INFO [INIT] Marking all workers as offline 2019-10-31 13:07:08,871 INFO [INIT] LAVA master has started. 2019-10-31 13:07:08,871 INFO [INIT] Using protocol version 3 2019-10-31 13:07:09,064 DEBUG Dhanu => PING(20) 2019-10-31 13:07:09,065 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:07:21,598 DEBUG lava-logs => PING(20) 2019-10-31 13:07:28,879 INFO scheduling health checks: 2019-10-31 13:07:28,905 INFO scheduling jobs: 2019-10-31 13:07:28,907 DEBUG - qemu 2019-10-31 13:07:28,979 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:07:28,979 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:07:28,983 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:07:30,080 INFO [INIT] Dropping privileges 2019-10-31 13:07:30,080 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:07:30,080 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:07:30,082 INFO [INIT] Marking all workers as offline 2019-10-31 13:07:30,107 INFO [INIT] LAVA master has started. 2019-10-31 13:07:30,107 INFO [INIT] Using protocol version 3 2019-10-31 13:07:30,124 DEBUG Dhanu => PING(20) 2019-10-31 13:07:30,124 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:07:41,612 DEBUG lava-logs => PING(20) 2019-10-31 13:07:50,116 INFO scheduling health checks: 2019-10-31 13:07:50,140 INFO scheduling jobs: 2019-10-31 13:07:50,142 DEBUG - qemu 2019-10-31 13:07:50,211 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:07:50,212 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:07:50,216 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:07:51,333 INFO [INIT] Dropping privileges 2019-10-31 13:07:51,333 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:07:51,333 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:07:51,335 INFO [INIT] Marking all workers as offline 2019-10-31 13:07:51,365 INFO [INIT] LAVA master has started. 2019-10-31 13:07:51,365 INFO [INIT] Using protocol version 3 2019-10-31 13:08:01,625 DEBUG lava-logs => PING(20) 2019-10-31 13:08:10,169 DEBUG Dhanu => PING(20) 2019-10-31 13:08:10,170 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:08:11,366 INFO scheduling health checks: 2019-10-31 13:08:11,398 INFO scheduling jobs: 2019-10-31 13:08:11,399 DEBUG - qemu 2019-10-31 13:08:11,469 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:08:11,469 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:08:11,474 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:08:12,602 INFO [INIT] Dropping privileges 2019-10-31 13:08:12,603 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:08:12,603 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:08:12,604 INFO [INIT] Marking all workers as offline 2019-10-31 13:08:12,668 INFO [INIT] LAVA master has started. 2019-10-31 13:08:12,668 INFO [INIT] Using protocol version 3 2019-10-31 13:08:21,644 DEBUG lava-logs => PING(20) 2019-10-31 13:08:30,191 DEBUG Dhanu => PING(20) 2019-10-31 13:08:30,191 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:08:32,669 INFO scheduling health checks: 2019-10-31 13:08:32,700 INFO scheduling jobs: 2019-10-31 13:08:32,702 DEBUG - qemu 2019-10-31 13:08:32,775 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:08:32,775 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:08:32,779 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:08:33,848 INFO [INIT] Dropping privileges 2019-10-31 13:08:33,848 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:08:33,848 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:08:33,850 INFO [INIT] Marking all workers as offline 2019-10-31 13:08:33,880 INFO [INIT] LAVA master has started. 2019-10-31 13:08:33,880 INFO [INIT] Using protocol version 3 2019-10-31 13:08:41,665 DEBUG lava-logs => PING(20) 2019-10-31 13:08:50,212 DEBUG Dhanu => PING(20) 2019-10-31 13:08:50,212 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:08:53,884 INFO scheduling health checks: 2019-10-31 13:08:53,936 INFO scheduling jobs: 2019-10-31 13:08:53,939 DEBUG - qemu 2019-10-31 13:08:54,006 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:08:54,006 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:08:54,011 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:08:55,094 INFO [INIT] Dropping privileges 2019-10-31 13:08:55,094 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:08:55,094 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:08:55,096 INFO [INIT] Marking all workers as offline 2019-10-31 13:08:55,127 INFO [INIT] LAVA master has started. 2019-10-31 13:08:55,127 INFO [INIT] Using protocol version 3 2019-10-31 13:09:01,684 DEBUG lava-logs => PING(20) 2019-10-31 13:09:10,235 DEBUG Dhanu => PING(20) 2019-10-31 13:09:10,236 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:09:15,131 INFO scheduling health checks: 2019-10-31 13:09:15,160 INFO scheduling jobs: 2019-10-31 13:09:15,162 DEBUG - qemu 2019-10-31 13:09:15,233 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:09:15,234 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:09:15,238 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:09:16,337 INFO [INIT] Dropping privileges 2019-10-31 13:09:16,337 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:09:16,337 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:09:16,339 INFO [INIT] Marking all workers as offline 2019-10-31 13:09:16,374 INFO [INIT] LAVA master has started. 2019-10-31 13:09:16,374 INFO [INIT] Using protocol version 3 2019-10-31 13:09:21,702 DEBUG lava-logs => PING(20) 2019-10-31 13:09:30,258 DEBUG Dhanu => PING(20) 2019-10-31 13:09:30,258 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:09:36,380 INFO scheduling health checks: 2019-10-31 13:09:36,418 INFO scheduling jobs: 2019-10-31 13:09:36,420 DEBUG - qemu 2019-10-31 13:09:36,490 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:09:36,490 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:09:36,494 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:09:37,596 INFO [INIT] Dropping privileges 2019-10-31 13:09:37,597 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:09:37,597 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:09:37,598 INFO [INIT] Marking all workers as offline 2019-10-31 13:09:37,631 INFO [INIT] LAVA master has started. 2019-10-31 13:09:37,631 INFO [INIT] Using protocol version 3 2019-10-31 13:09:41,718 DEBUG lava-logs => PING(20) 2019-10-31 13:09:50,277 DEBUG Dhanu => PING(20) 2019-10-31 13:09:50,277 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:09:57,638 INFO scheduling health checks: 2019-10-31 13:09:57,676 INFO scheduling jobs: 2019-10-31 13:09:57,679 DEBUG - qemu 2019-10-31 13:09:57,754 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:09:57,754 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:09:57,758 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:09:58,879 INFO [INIT] Dropping privileges 2019-10-31 13:09:58,880 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:09:58,880 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:09:58,881 INFO [INIT] Marking all workers as offline 2019-10-31 13:09:58,945 INFO [INIT] LAVA master has started. 2019-10-31 13:09:58,945 INFO [INIT] Using protocol version 3 2019-10-31 13:10:01,733 DEBUG lava-logs => PING(20) 2019-10-31 13:10:10,297 DEBUG Dhanu => PING(20) 2019-10-31 13:10:10,298 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:10:18,954 INFO scheduling health checks: 2019-10-31 13:10:18,989 INFO scheduling jobs: 2019-10-31 13:10:18,991 DEBUG - qemu 2019-10-31 13:10:19,063 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:10:19,063 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:10:19,067 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:10:20,333 INFO [INIT] Dropping privileges 2019-10-31 13:10:20,333 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:10:20,333 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:10:20,335 INFO [INIT] Marking all workers as offline 2019-10-31 13:10:20,402 INFO [INIT] LAVA master has started. 2019-10-31 13:10:20,402 INFO [INIT] Using protocol version 3 2019-10-31 13:10:21,746 DEBUG lava-logs => PING(20) 2019-10-31 13:10:30,322 DEBUG Dhanu => PING(20) 2019-10-31 13:10:30,322 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:10:40,413 INFO scheduling health checks: 2019-10-31 13:10:40,446 INFO scheduling jobs: 2019-10-31 13:10:40,448 DEBUG - qemu 2019-10-31 13:10:40,520 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:10:40,520 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:10:40,525 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:10:41,607 INFO [INIT] Dropping privileges 2019-10-31 13:10:41,607 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:10:41,607 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:10:41,609 INFO [INIT] Marking all workers as offline 2019-10-31 13:10:41,671 INFO [INIT] LAVA master has started. 2019-10-31 13:10:41,671 INFO [INIT] Using protocol version 3 2019-10-31 13:10:41,759 DEBUG lava-logs => PING(20) 2019-10-31 13:10:50,340 DEBUG Dhanu => PING(20) 2019-10-31 13:10:50,340 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:11:01,683 INFO scheduling health checks: 2019-10-31 13:11:01,715 INFO scheduling jobs: 2019-10-31 13:11:01,717 DEBUG - qemu 2019-10-31 13:11:01,784 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:11:01,784 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:11:01,795 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:11:03,097 INFO [INIT] Dropping privileges 2019-10-31 13:11:03,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:11:03,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:11:03,099 INFO [INIT] Marking all workers as offline 2019-10-31 13:11:03,128 INFO [INIT] LAVA master has started. 2019-10-31 13:11:03,128 INFO [INIT] Using protocol version 3 2019-10-31 13:11:10,361 DEBUG Dhanu => PING(20) 2019-10-31 13:11:10,362 WARNING Unknown dispatcher (server crashed) 2019-10-31 13:11:21,794 DEBUG lava-logs => PING(20) 2019-10-31 13:11:23,130 INFO scheduling health checks: 2019-10-31 13:11:23,150 INFO scheduling jobs: 2019-10-31 13:11:23,151 DEBUG - qemu 2019-10-31 13:11:23,216 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 13:11:23,216 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 13:11:23,221 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 13:11:24,341 INFO [INIT] Dropping privileges 2019-10-31 13:11:24,341 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 13:11:24,341 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 13:11:24,343 INFO [INIT] Marking all workers as offline 2019-10-31 13:11:24,376 INFO [INIT] LAVA master has started. 2019-10-31 13:11:24,376 INFO [INIT] Using protocol version 3 2019-10-31 13:11:30,381 DEBUG Dhanu => PING(20) 2019-10-31 13:11:30,382 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:00:08,991 DEBUG Dhanu => PING(20) 2019-10-31 14:00:09,036 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 14:00:15,416 DEBUG lava-logs => PING(20) 2019-10-31 14:00:29,008 DEBUG Dhanu => PING(20) 2019-10-31 14:00:29,042 INFO scheduling health checks: 2019-10-31 14:00:29,071 INFO scheduling jobs: 2019-10-31 14:00:29,073 DEBUG - qemu 2019-10-31 14:00:29,171 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:00:29,171 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:00:29,177 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:00:30,610 INFO [INIT] Dropping privileges 2019-10-31 14:00:30,610 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:00:30,610 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:00:30,612 INFO [INIT] Marking all workers as offline 2019-10-31 14:00:30,642 INFO [INIT] LAVA master has started. 2019-10-31 14:00:30,643 INFO [INIT] Using protocol version 3 2019-10-31 14:00:35,433 DEBUG lava-logs => PING(20) 2019-10-31 14:00:49,026 DEBUG Dhanu => PING(20) 2019-10-31 14:00:49,026 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:00:50,645 INFO scheduling health checks: 2019-10-31 14:00:50,740 INFO scheduling jobs: 2019-10-31 14:00:50,743 DEBUG - qemu 2019-10-31 14:00:50,821 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:00:50,821 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:00:50,825 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:00:52,077 INFO [INIT] Dropping privileges 2019-10-31 14:00:52,078 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:00:52,078 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:00:52,080 INFO [INIT] Marking all workers as offline 2019-10-31 14:00:52,108 INFO [INIT] LAVA master has started. 2019-10-31 14:00:52,109 INFO [INIT] Using protocol version 3 2019-10-31 14:00:55,448 DEBUG lava-logs => PING(20) 2019-10-31 14:01:09,042 DEBUG Dhanu => PING(20) 2019-10-31 14:01:09,042 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:01:12,111 INFO scheduling health checks: 2019-10-31 14:01:12,143 INFO scheduling jobs: 2019-10-31 14:01:12,146 DEBUG - qemu 2019-10-31 14:01:12,241 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:01:12,241 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:01:12,250 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:01:13,659 INFO [INIT] Dropping privileges 2019-10-31 14:01:13,659 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:01:13,660 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:01:13,661 INFO [INIT] Marking all workers as offline 2019-10-31 14:01:13,722 INFO [INIT] LAVA master has started. 2019-10-31 14:01:13,722 INFO [INIT] Using protocol version 3 2019-10-31 14:01:15,463 DEBUG lava-logs => PING(20) 2019-10-31 14:01:29,056 DEBUG Dhanu => PING(20) 2019-10-31 14:01:29,056 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:01:33,723 INFO scheduling health checks: 2019-10-31 14:01:33,766 INFO scheduling jobs: 2019-10-31 14:01:33,769 DEBUG - qemu 2019-10-31 14:01:33,842 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:01:33,842 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:01:33,847 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:01:35,324 INFO [INIT] Dropping privileges 2019-10-31 14:01:35,324 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:01:35,325 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:01:35,326 INFO [INIT] Marking all workers as offline 2019-10-31 14:01:35,380 INFO [INIT] LAVA master has started. 2019-10-31 14:01:35,380 INFO [INIT] Using protocol version 3 2019-10-31 14:01:35,472 DEBUG lava-logs => PING(20) 2019-10-31 14:01:49,072 DEBUG Dhanu => PING(20) 2019-10-31 14:01:49,072 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:01:55,387 INFO scheduling health checks: 2019-10-31 14:01:55,423 INFO scheduling jobs: 2019-10-31 14:01:55,426 DEBUG - qemu 2019-10-31 14:01:55,504 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:01:55,504 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:01:55,511 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:01:57,034 INFO [INIT] Dropping privileges 2019-10-31 14:01:57,035 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:01:57,035 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:01:57,037 INFO [INIT] Marking all workers as offline 2019-10-31 14:01:57,071 INFO [INIT] LAVA master has started. 2019-10-31 14:01:57,071 INFO [INIT] Using protocol version 3 2019-10-31 14:02:09,090 DEBUG Dhanu => PING(20) 2019-10-31 14:02:09,090 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:02:15,508 DEBUG lava-logs => PING(20) 2019-10-31 14:02:17,073 INFO scheduling health checks: 2019-10-31 14:02:17,092 INFO scheduling jobs: 2019-10-31 14:02:17,095 DEBUG - qemu 2019-10-31 14:02:17,175 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:02:17,175 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:02:17,184 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:02:18,746 INFO [INIT] Dropping privileges 2019-10-31 14:02:18,746 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:02:18,746 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:02:18,748 INFO [INIT] Marking all workers as offline 2019-10-31 14:02:18,784 INFO [INIT] LAVA master has started. 2019-10-31 14:02:18,784 INFO [INIT] Using protocol version 3 2019-10-31 14:02:29,104 DEBUG Dhanu => PING(20) 2019-10-31 14:02:29,104 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:02:35,524 DEBUG lava-logs => PING(20) 2019-10-31 14:02:38,787 INFO scheduling health checks: 2019-10-31 14:02:38,816 INFO scheduling jobs: 2019-10-31 14:02:38,818 DEBUG - qemu 2019-10-31 14:02:38,903 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 14:02:38,903 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 14:02:38,908 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:02:40,444 INFO [INIT] Dropping privileges 2019-10-31 14:02:40,444 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:02:40,448 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:02:40,450 INFO [INIT] Marking all workers as offline 2019-10-31 14:02:40,564 INFO [INIT] LAVA master has started. 2019-10-31 14:02:40,564 INFO [INIT] Using protocol version 3 2019-10-31 14:02:49,122 DEBUG Dhanu => PING(20) 2019-10-31 14:02:49,123 WARNING Unknown dispatcher (server crashed) 2019-10-31 14:02:55,540 DEBUG lava-logs => PING(20) 2019-10-31 14:03:00,571 INFO scheduling health checks: 2019-10-31 14:03:00,577 INFO scheduling jobs: 2019-10-31 14:03:09,144 DEBUG Dhanu => PING(20) 2019-10-31 14:03:15,560 DEBUG lava-logs => PING(20) 2019-10-31 14:03:20,592 INFO scheduling health checks: 2019-10-31 14:03:20,596 INFO scheduling jobs: 2019-10-31 14:03:29,160 DEBUG Dhanu => PING(20) 2019-10-31 14:03:35,583 DEBUG lava-logs => PING(20) 2019-10-31 14:03:40,611 INFO scheduling health checks: 2019-10-31 14:03:40,613 INFO scheduling jobs: 2019-10-31 14:03:49,180 DEBUG Dhanu => PING(20) 2019-10-31 14:03:55,600 DEBUG lava-logs => PING(20) 2019-10-31 14:04:00,629 INFO scheduling health checks: 2019-10-31 14:04:00,633 INFO scheduling jobs: 2019-10-31 14:04:09,199 DEBUG Dhanu => PING(20) 2019-10-31 14:04:15,620 DEBUG lava-logs => PING(20) 2019-10-31 14:04:20,650 INFO scheduling health checks: 2019-10-31 14:04:20,654 INFO scheduling jobs: 2019-10-31 14:04:29,224 DEBUG Dhanu => PING(20) 2019-10-31 14:04:35,643 DEBUG lava-logs => PING(20) 2019-10-31 14:04:40,668 INFO scheduling health checks: 2019-10-31 14:04:40,672 INFO scheduling jobs: 2019-10-31 14:04:49,248 DEBUG Dhanu => PING(20) 2019-10-31 14:04:55,666 DEBUG lava-logs => PING(20) 2019-10-31 14:05:00,684 INFO scheduling health checks: 2019-10-31 14:05:00,688 INFO scheduling jobs: 2019-10-31 14:05:09,273 DEBUG Dhanu => PING(20) 2019-10-31 14:05:15,684 DEBUG lava-logs => PING(20) 2019-10-31 14:05:20,705 INFO scheduling health checks: 2019-10-31 14:05:20,709 INFO scheduling jobs: 2019-10-31 14:05:29,297 DEBUG Dhanu => PING(20) 2019-10-31 14:05:35,707 DEBUG lava-logs => PING(20) 2019-10-31 14:05:40,725 INFO scheduling health checks: 2019-10-31 14:05:40,729 INFO scheduling jobs: 2019-10-31 14:05:49,322 DEBUG Dhanu => PING(20) 2019-10-31 14:05:55,730 DEBUG lava-logs => PING(20) 2019-10-31 14:06:00,743 INFO scheduling health checks: 2019-10-31 14:06:00,747 INFO scheduling jobs: 2019-10-31 14:06:09,346 DEBUG Dhanu => PING(20) 2019-10-31 14:06:15,753 DEBUG lava-logs => PING(20) 2019-10-31 14:06:20,762 INFO scheduling health checks: 2019-10-31 14:06:20,766 INFO scheduling jobs: 2019-10-31 14:06:29,370 DEBUG Dhanu => PING(20) 2019-10-31 14:06:35,776 DEBUG lava-logs => PING(20) 2019-10-31 14:06:40,781 INFO scheduling health checks: 2019-10-31 14:06:40,785 INFO scheduling jobs: 2019-10-31 14:06:49,391 DEBUG Dhanu => PING(20) 2019-10-31 14:06:55,799 DEBUG lava-logs => PING(20) 2019-10-31 14:07:00,798 INFO scheduling health checks: 2019-10-31 14:07:00,803 INFO scheduling jobs: 2019-10-31 14:07:09,412 DEBUG Dhanu => PING(20) 2019-10-31 14:07:15,822 DEBUG lava-logs => PING(20) 2019-10-31 14:07:20,818 INFO scheduling health checks: 2019-10-31 14:07:20,822 INFO scheduling jobs: 2019-10-31 14:07:29,437 DEBUG Dhanu => PING(20) 2019-10-31 14:07:35,845 DEBUG lava-logs => PING(20) 2019-10-31 14:07:40,836 INFO scheduling health checks: 2019-10-31 14:07:40,841 INFO scheduling jobs: 2019-10-31 14:07:49,466 DEBUG Dhanu => PING(20) 2019-10-31 14:07:55,867 DEBUG lava-logs => PING(20) 2019-10-31 14:08:00,855 INFO scheduling health checks: 2019-10-31 14:08:00,859 INFO scheduling jobs: 2019-10-31 14:08:09,486 DEBUG Dhanu => PING(20) 2019-10-31 14:08:15,890 DEBUG lava-logs => PING(20) 2019-10-31 14:08:20,876 INFO scheduling health checks: 2019-10-31 14:08:20,880 INFO scheduling jobs: 2019-10-31 14:08:29,510 DEBUG Dhanu => PING(20) 2019-10-31 14:08:35,913 DEBUG lava-logs => PING(20) 2019-10-31 14:08:40,898 INFO scheduling health checks: 2019-10-31 14:08:40,902 INFO scheduling jobs: 2019-10-31 14:08:49,535 DEBUG Dhanu => PING(20) 2019-10-31 14:08:55,936 DEBUG lava-logs => PING(20) 2019-10-31 14:09:00,916 INFO scheduling health checks: 2019-10-31 14:09:00,920 INFO scheduling jobs: 2019-10-31 14:09:09,552 DEBUG Dhanu => PING(20) 2019-10-31 14:09:15,959 DEBUG lava-logs => PING(20) 2019-10-31 14:09:20,935 INFO scheduling health checks: 2019-10-31 14:09:20,939 INFO scheduling jobs: 2019-10-31 14:09:29,575 DEBUG Dhanu => PING(20) 2019-10-31 14:09:35,980 DEBUG lava-logs => PING(20) 2019-10-31 14:09:40,953 INFO scheduling health checks: 2019-10-31 14:09:40,957 INFO scheduling jobs: 2019-10-31 14:09:49,599 DEBUG Dhanu => PING(20) 2019-10-31 14:09:56,003 DEBUG lava-logs => PING(20) 2019-10-31 14:10:00,973 INFO scheduling health checks: 2019-10-31 14:10:00,977 INFO scheduling jobs: 2019-10-31 14:10:09,624 DEBUG Dhanu => PING(20) 2019-10-31 14:10:16,026 DEBUG lava-logs => PING(20) 2019-10-31 14:10:20,996 INFO scheduling health checks: 2019-10-31 14:10:20,999 INFO scheduling jobs: 2019-10-31 14:10:29,644 DEBUG Dhanu => PING(20) 2019-10-31 14:10:36,049 DEBUG lava-logs => PING(20) 2019-10-31 14:10:41,012 INFO scheduling health checks: 2019-10-31 14:10:41,016 INFO scheduling jobs: 2019-10-31 14:10:49,668 DEBUG Dhanu => PING(20) 2019-10-31 14:10:56,068 DEBUG lava-logs => PING(20) 2019-10-31 14:11:01,029 INFO scheduling health checks: 2019-10-31 14:11:01,031 INFO scheduling jobs: 2019-10-31 14:11:09,693 DEBUG Dhanu => PING(20) 2019-10-31 14:11:16,088 DEBUG lava-logs => PING(20) 2019-10-31 14:11:21,049 INFO scheduling health checks: 2019-10-31 14:11:21,054 INFO scheduling jobs: 2019-10-31 14:11:29,712 DEBUG Dhanu => PING(20) 2019-10-31 14:11:36,107 DEBUG lava-logs => PING(20) 2019-10-31 14:11:41,071 INFO scheduling health checks: 2019-10-31 14:11:41,076 INFO scheduling jobs: 2019-10-31 14:11:49,735 DEBUG Dhanu => PING(20) 2019-10-31 14:11:56,130 DEBUG lava-logs => PING(20) 2019-10-31 14:12:01,092 INFO scheduling health checks: 2019-10-31 14:12:01,095 INFO scheduling jobs: 2019-10-31 14:12:09,756 DEBUG Dhanu => PING(20) 2019-10-31 14:12:16,151 DEBUG lava-logs => PING(20) 2019-10-31 14:12:21,110 INFO scheduling health checks: 2019-10-31 14:12:21,114 INFO scheduling jobs: 2019-10-31 14:12:29,777 DEBUG Dhanu => PING(20) 2019-10-31 14:12:36,170 DEBUG lava-logs => PING(20) 2019-10-31 14:12:41,131 INFO scheduling health checks: 2019-10-31 14:12:41,135 INFO scheduling jobs: 2019-10-31 14:12:49,798 DEBUG Dhanu => PING(20) 2019-10-31 14:12:56,190 DEBUG lava-logs => PING(20) 2019-10-31 14:13:01,149 INFO scheduling health checks: 2019-10-31 14:13:01,151 INFO scheduling jobs: 2019-10-31 14:13:09,816 DEBUG Dhanu => PING(20) 2019-10-31 14:13:16,208 DEBUG lava-logs => PING(20) 2019-10-31 14:13:21,165 INFO scheduling health checks: 2019-10-31 14:13:21,170 INFO scheduling jobs: 2019-10-31 14:13:29,832 DEBUG Dhanu => PING(20) 2019-10-31 14:13:36,228 DEBUG lava-logs => PING(20) 2019-10-31 14:13:41,182 INFO scheduling health checks: 2019-10-31 14:13:41,184 INFO scheduling jobs: 2019-10-31 14:13:49,852 DEBUG Dhanu => PING(20) 2019-10-31 14:13:56,251 DEBUG lava-logs => PING(20) 2019-10-31 14:14:01,204 INFO scheduling health checks: 2019-10-31 14:14:01,208 INFO scheduling jobs: 2019-10-31 14:14:09,867 DEBUG Dhanu => PING(20) 2019-10-31 14:14:16,268 DEBUG lava-logs => PING(20) 2019-10-31 14:14:21,227 INFO scheduling health checks: 2019-10-31 14:14:21,229 INFO scheduling jobs: 2019-10-31 14:14:29,886 DEBUG Dhanu => PING(20) 2019-10-31 14:14:36,288 DEBUG lava-logs => PING(20) 2019-10-31 14:14:41,239 INFO scheduling health checks: 2019-10-31 14:14:41,242 INFO scheduling jobs: 2019-10-31 14:14:49,905 DEBUG Dhanu => PING(20) 2019-10-31 14:14:56,307 DEBUG lava-logs => PING(20) 2019-10-31 14:15:01,256 INFO scheduling health checks: 2019-10-31 14:15:01,260 INFO scheduling jobs: 2019-10-31 14:15:09,924 DEBUG Dhanu => PING(20) 2019-10-31 14:15:16,326 DEBUG lava-logs => PING(20) 2019-10-31 14:15:21,274 INFO scheduling health checks: 2019-10-31 14:15:21,278 INFO scheduling jobs: 2019-10-31 14:15:29,946 DEBUG Dhanu => PING(20) 2019-10-31 14:15:36,347 DEBUG lava-logs => PING(20) 2019-10-31 14:15:41,294 INFO scheduling health checks: 2019-10-31 14:15:41,297 INFO scheduling jobs: 2019-10-31 14:15:49,966 DEBUG Dhanu => PING(20) 2019-10-31 14:15:56,364 DEBUG lava-logs => PING(20) 2019-10-31 14:16:01,313 INFO scheduling health checks: 2019-10-31 14:16:01,316 INFO scheduling jobs: 2019-10-31 14:16:09,986 DEBUG Dhanu => PING(20) 2019-10-31 14:16:16,380 DEBUG lava-logs => PING(20) 2019-10-31 14:16:21,328 INFO scheduling health checks: 2019-10-31 14:16:21,332 INFO scheduling jobs: 2019-10-31 14:16:30,004 DEBUG Dhanu => PING(20) 2019-10-31 14:16:36,401 DEBUG lava-logs => PING(20) 2019-10-31 14:16:41,346 INFO scheduling health checks: 2019-10-31 14:16:41,350 INFO scheduling jobs: 2019-10-31 14:16:50,027 DEBUG Dhanu => PING(20) 2019-10-31 14:16:56,424 DEBUG lava-logs => PING(20) 2019-10-31 14:17:01,364 INFO scheduling health checks: 2019-10-31 14:17:01,368 INFO scheduling jobs: 2019-10-31 14:17:10,046 DEBUG Dhanu => PING(20) 2019-10-31 14:17:16,447 DEBUG lava-logs => PING(20) 2019-10-31 14:17:21,385 INFO scheduling health checks: 2019-10-31 14:17:21,389 INFO scheduling jobs: 2019-10-31 14:17:30,070 DEBUG Dhanu => PING(20) 2019-10-31 14:17:36,460 DEBUG lava-logs => PING(20) 2019-10-31 14:17:41,403 INFO scheduling health checks: 2019-10-31 14:17:41,407 INFO scheduling jobs: 2019-10-31 14:17:50,094 DEBUG Dhanu => PING(20) 2019-10-31 14:17:56,483 DEBUG lava-logs => PING(20) 2019-10-31 14:18:01,421 INFO scheduling health checks: 2019-10-31 14:18:01,425 INFO scheduling jobs: 2019-10-31 14:18:10,112 DEBUG Dhanu => PING(20) 2019-10-31 14:18:16,506 DEBUG lava-logs => PING(20) 2019-10-31 14:18:21,442 INFO scheduling health checks: 2019-10-31 14:18:21,446 INFO scheduling jobs: 2019-10-31 14:18:30,135 DEBUG Dhanu => PING(20) 2019-10-31 14:18:36,529 DEBUG lava-logs => PING(20) 2019-10-31 14:18:41,461 INFO scheduling health checks: 2019-10-31 14:18:41,465 INFO scheduling jobs: 2019-10-31 14:18:50,160 DEBUG Dhanu => PING(20) 2019-10-31 14:18:56,552 DEBUG lava-logs => PING(20) 2019-10-31 14:19:01,481 INFO scheduling health checks: 2019-10-31 14:19:01,485 INFO scheduling jobs: 2019-10-31 14:19:10,184 DEBUG Dhanu => PING(20) 2019-10-31 14:19:16,575 DEBUG lava-logs => PING(20) 2019-10-31 14:19:21,500 INFO scheduling health checks: 2019-10-31 14:19:21,504 INFO scheduling jobs: 2019-10-31 14:19:30,209 DEBUG Dhanu => PING(20) 2019-10-31 14:19:36,594 DEBUG lava-logs => PING(20) 2019-10-31 14:19:41,516 INFO scheduling health checks: 2019-10-31 14:19:41,520 INFO scheduling jobs: 2019-10-31 14:19:50,233 DEBUG Dhanu => PING(20) 2019-10-31 14:19:56,617 DEBUG lava-logs => PING(20) 2019-10-31 14:20:01,536 INFO scheduling health checks: 2019-10-31 14:20:01,540 INFO scheduling jobs: 2019-10-31 14:20:10,254 DEBUG Dhanu => PING(20) 2019-10-31 14:20:16,640 DEBUG lava-logs => PING(20) 2019-10-31 14:20:21,553 INFO scheduling health checks: 2019-10-31 14:20:21,557 INFO scheduling jobs: 2019-10-31 14:20:30,278 DEBUG Dhanu => PING(20) 2019-10-31 14:20:36,663 DEBUG lava-logs => PING(20) 2019-10-31 14:20:41,571 INFO scheduling health checks: 2019-10-31 14:20:41,575 INFO scheduling jobs: 2019-10-31 14:20:50,303 DEBUG Dhanu => PING(20) 2019-10-31 14:20:56,684 DEBUG lava-logs => PING(20) 2019-10-31 14:21:01,589 INFO scheduling health checks: 2019-10-31 14:21:01,593 INFO scheduling jobs: 2019-10-31 14:21:10,327 DEBUG Dhanu => PING(20) 2019-10-31 14:21:16,707 DEBUG lava-logs => PING(20) 2019-10-31 14:21:21,608 INFO scheduling health checks: 2019-10-31 14:21:21,612 INFO scheduling jobs: 2019-10-31 14:21:30,350 DEBUG Dhanu => PING(20) 2019-10-31 14:21:36,724 DEBUG lava-logs => PING(20) 2019-10-31 14:21:41,626 INFO scheduling health checks: 2019-10-31 14:21:41,630 INFO scheduling jobs: 2019-10-31 14:21:50,370 DEBUG Dhanu => PING(20) 2019-10-31 14:21:56,744 DEBUG lava-logs => PING(20) 2019-10-31 14:22:01,646 INFO scheduling health checks: 2019-10-31 14:22:01,650 INFO scheduling jobs: 2019-10-31 14:22:10,394 DEBUG Dhanu => PING(20) 2019-10-31 14:22:16,767 DEBUG lava-logs => PING(20) 2019-10-31 14:22:21,664 INFO scheduling health checks: 2019-10-31 14:22:21,668 INFO scheduling jobs: 2019-10-31 14:22:30,415 DEBUG Dhanu => PING(20) 2019-10-31 14:22:36,780 DEBUG lava-logs => PING(20) 2019-10-31 14:22:41,686 INFO scheduling health checks: 2019-10-31 14:22:41,690 INFO scheduling jobs: 2019-10-31 14:22:50,440 DEBUG Dhanu => PING(20) 2019-10-31 14:22:56,800 DEBUG lava-logs => PING(20) 2019-10-31 14:23:01,706 INFO scheduling health checks: 2019-10-31 14:23:01,710 INFO scheduling jobs: 2019-10-31 14:23:10,462 DEBUG Dhanu => PING(20) 2019-10-31 14:23:16,823 DEBUG lava-logs => PING(20) 2019-10-31 14:23:21,724 INFO scheduling health checks: 2019-10-31 14:23:21,728 INFO scheduling jobs: 2019-10-31 14:23:30,486 DEBUG Dhanu => PING(20) 2019-10-31 14:23:36,846 DEBUG lava-logs => PING(20) 2019-10-31 14:23:41,743 INFO scheduling health checks: 2019-10-31 14:23:41,747 INFO scheduling jobs: 2019-10-31 14:23:50,510 DEBUG Dhanu => PING(20) 2019-10-31 14:23:56,869 DEBUG lava-logs => PING(20) 2019-10-31 14:24:01,762 INFO scheduling health checks: 2019-10-31 14:24:01,766 INFO scheduling jobs: 2019-10-31 14:24:10,528 DEBUG Dhanu => PING(20) 2019-10-31 14:24:16,888 DEBUG lava-logs => PING(20) 2019-10-31 14:24:21,780 INFO scheduling health checks: 2019-10-31 14:24:21,784 INFO scheduling jobs: 2019-10-31 14:24:30,544 DEBUG Dhanu => PING(20) 2019-10-31 14:24:36,905 DEBUG lava-logs => PING(20) 2019-10-31 14:24:41,801 INFO scheduling health checks: 2019-10-31 14:24:41,805 INFO scheduling jobs: 2019-10-31 14:24:50,560 DEBUG Dhanu => PING(20) 2019-10-31 14:24:56,928 DEBUG lava-logs => PING(20) 2019-10-31 14:25:01,819 INFO scheduling health checks: 2019-10-31 14:25:01,823 INFO scheduling jobs: 2019-10-31 14:25:10,576 DEBUG Dhanu => PING(20) 2019-10-31 14:25:16,951 DEBUG lava-logs => PING(20) 2019-10-31 14:25:21,838 INFO scheduling health checks: 2019-10-31 14:25:21,842 INFO scheduling jobs: 2019-10-31 14:25:30,600 DEBUG Dhanu => PING(20) 2019-10-31 14:25:36,974 DEBUG lava-logs => PING(20) 2019-10-31 14:25:41,858 INFO scheduling health checks: 2019-10-31 14:25:41,862 INFO scheduling jobs: 2019-10-31 14:25:50,625 DEBUG Dhanu => PING(20) 2019-10-31 14:25:56,994 DEBUG lava-logs => PING(20) 2019-10-31 14:26:01,877 INFO scheduling health checks: 2019-10-31 14:26:01,881 INFO scheduling jobs: 2019-10-31 14:26:10,649 DEBUG Dhanu => PING(20) 2019-10-31 14:26:17,014 DEBUG lava-logs => PING(20) 2019-10-31 14:26:21,900 INFO scheduling health checks: 2019-10-31 14:26:21,904 INFO scheduling jobs: 2019-10-31 14:26:30,672 DEBUG Dhanu => PING(20) 2019-10-31 14:26:37,036 DEBUG lava-logs => PING(20) 2019-10-31 14:26:41,921 INFO scheduling health checks: 2019-10-31 14:26:41,925 INFO scheduling jobs: 2019-10-31 14:26:50,694 DEBUG Dhanu => PING(20) 2019-10-31 14:26:57,056 DEBUG lava-logs => PING(20) 2019-10-31 14:27:01,940 INFO scheduling health checks: 2019-10-31 14:27:01,944 INFO scheduling jobs: 2019-10-31 14:27:10,718 DEBUG Dhanu => PING(20) 2019-10-31 14:27:17,079 DEBUG lava-logs => PING(20) 2019-10-31 14:27:21,962 INFO scheduling health checks: 2019-10-31 14:27:21,966 INFO scheduling jobs: 2019-10-31 14:27:30,741 DEBUG Dhanu => PING(20) 2019-10-31 14:27:37,102 DEBUG lava-logs => PING(20) 2019-10-31 14:27:41,981 INFO scheduling health checks: 2019-10-31 14:27:41,985 INFO scheduling jobs: 2019-10-31 14:27:50,765 DEBUG Dhanu => PING(20) 2019-10-31 14:27:57,125 DEBUG lava-logs => PING(20) 2019-10-31 14:28:01,997 INFO scheduling health checks: 2019-10-31 14:28:02,000 INFO scheduling jobs: 2019-10-31 14:28:10,789 DEBUG Dhanu => PING(20) 2019-10-31 14:28:17,148 DEBUG lava-logs => PING(20) 2019-10-31 14:28:22,017 INFO scheduling health checks: 2019-10-31 14:28:22,021 INFO scheduling jobs: 2019-10-31 14:28:30,814 DEBUG Dhanu => PING(20) 2019-10-31 14:28:37,163 DEBUG lava-logs => PING(20) 2019-10-31 14:28:42,035 INFO scheduling health checks: 2019-10-31 14:28:42,039 INFO scheduling jobs: 2019-10-31 14:28:50,838 DEBUG Dhanu => PING(20) 2019-10-31 14:28:57,186 DEBUG lava-logs => PING(20) 2019-10-31 14:29:02,057 INFO scheduling health checks: 2019-10-31 14:29:02,061 INFO scheduling jobs: 2019-10-31 14:29:10,862 DEBUG Dhanu => PING(20) 2019-10-31 14:29:17,209 DEBUG lava-logs => PING(20) 2019-10-31 14:29:22,077 INFO scheduling health checks: 2019-10-31 14:29:22,081 INFO scheduling jobs: 2019-10-31 14:29:30,883 DEBUG Dhanu => PING(20) 2019-10-31 14:29:37,228 DEBUG lava-logs => PING(20) 2019-10-31 14:29:42,099 INFO scheduling health checks: 2019-10-31 14:29:42,102 INFO scheduling jobs: 2019-10-31 14:29:50,906 DEBUG Dhanu => PING(20) 2019-10-31 14:29:57,251 DEBUG lava-logs => PING(20) 2019-10-31 14:30:02,118 INFO scheduling health checks: 2019-10-31 14:30:02,121 INFO scheduling jobs: 2019-10-31 14:30:10,930 DEBUG Dhanu => PING(20) 2019-10-31 14:30:17,274 DEBUG lava-logs => PING(20) 2019-10-31 14:30:22,135 INFO scheduling health checks: 2019-10-31 14:30:22,138 INFO scheduling jobs: 2019-10-31 14:30:30,954 DEBUG Dhanu => PING(20) 2019-10-31 14:30:37,297 DEBUG lava-logs => PING(20) 2019-10-31 14:30:42,155 INFO scheduling health checks: 2019-10-31 14:30:42,159 INFO scheduling jobs: 2019-10-31 14:30:50,979 DEBUG Dhanu => PING(20) 2019-10-31 14:30:57,320 DEBUG lava-logs => PING(20) 2019-10-31 14:31:02,176 INFO scheduling health checks: 2019-10-31 14:31:02,180 INFO scheduling jobs: 2019-10-31 14:31:11,003 DEBUG Dhanu => PING(20) 2019-10-31 14:31:17,343 DEBUG lava-logs => PING(20) 2019-10-31 14:31:22,195 INFO scheduling health checks: 2019-10-31 14:31:22,199 INFO scheduling jobs: 2019-10-31 14:31:31,028 DEBUG Dhanu => PING(20) 2019-10-31 14:31:37,360 DEBUG lava-logs => PING(20) 2019-10-31 14:31:42,214 INFO scheduling health checks: 2019-10-31 14:31:42,218 INFO scheduling jobs: 2019-10-31 14:31:51,052 DEBUG Dhanu => PING(20) 2019-10-31 14:31:57,383 DEBUG lava-logs => PING(20) 2019-10-31 14:32:02,234 INFO scheduling health checks: 2019-10-31 14:32:02,238 INFO scheduling jobs: 2019-10-31 14:32:11,077 DEBUG Dhanu => PING(20) 2019-10-31 14:32:17,406 DEBUG lava-logs => PING(20) 2019-10-31 14:32:22,253 INFO scheduling health checks: 2019-10-31 14:32:22,257 INFO scheduling jobs: 2019-10-31 14:32:31,101 DEBUG Dhanu => PING(20) 2019-10-31 14:32:37,424 DEBUG lava-logs => PING(20) 2019-10-31 14:32:42,274 INFO scheduling health checks: 2019-10-31 14:32:42,278 INFO scheduling jobs: 2019-10-31 14:32:51,122 DEBUG Dhanu => PING(20) 2019-10-31 14:32:57,447 DEBUG lava-logs => PING(20) 2019-10-31 14:33:02,293 INFO scheduling health checks: 2019-10-31 14:33:02,297 INFO scheduling jobs: 2019-10-31 14:33:11,146 DEBUG Dhanu => PING(20) 2019-10-31 14:33:17,464 DEBUG lava-logs => PING(20) 2019-10-31 14:33:22,314 INFO scheduling health checks: 2019-10-31 14:33:22,317 INFO scheduling jobs: 2019-10-31 14:33:31,171 DEBUG Dhanu => PING(20) 2019-10-31 14:33:37,486 DEBUG lava-logs => PING(20) 2019-10-31 14:33:42,335 INFO scheduling health checks: 2019-10-31 14:33:42,338 INFO scheduling jobs: 2019-10-31 14:33:51,195 DEBUG Dhanu => PING(20) 2019-10-31 14:33:57,509 DEBUG lava-logs => PING(20) 2019-10-31 14:34:02,351 INFO scheduling health checks: 2019-10-31 14:34:02,354 INFO scheduling jobs: 2019-10-31 14:34:11,220 DEBUG Dhanu => PING(20) 2019-10-31 14:34:17,532 DEBUG lava-logs => PING(20) 2019-10-31 14:34:22,368 INFO scheduling health checks: 2019-10-31 14:34:22,372 INFO scheduling jobs: 2019-10-31 14:34:31,240 DEBUG Dhanu => PING(20) 2019-10-31 14:34:37,555 DEBUG lava-logs => PING(20) 2019-10-31 14:34:42,389 INFO scheduling health checks: 2019-10-31 14:34:42,393 INFO scheduling jobs: 2019-10-31 14:34:51,264 DEBUG Dhanu => PING(20) 2019-10-31 14:34:57,578 DEBUG lava-logs => PING(20) 2019-10-31 14:35:02,408 INFO scheduling health checks: 2019-10-31 14:35:02,412 INFO scheduling jobs: 2019-10-31 14:35:11,289 DEBUG Dhanu => PING(20) 2019-10-31 14:35:17,601 DEBUG lava-logs => PING(20) 2019-10-31 14:35:22,431 INFO scheduling health checks: 2019-10-31 14:35:22,435 INFO scheduling jobs: 2019-10-31 14:35:31,313 DEBUG Dhanu => PING(20) 2019-10-31 14:35:37,624 DEBUG lava-logs => PING(20) 2019-10-31 14:35:42,454 INFO scheduling health checks: 2019-10-31 14:35:42,458 INFO scheduling jobs: 2019-10-31 14:35:51,337 DEBUG Dhanu => PING(20) 2019-10-31 14:35:57,644 DEBUG lava-logs => PING(20) 2019-10-31 14:36:02,472 INFO scheduling health checks: 2019-10-31 14:36:02,476 INFO scheduling jobs: 2019-10-31 14:36:11,362 DEBUG Dhanu => PING(20) 2019-10-31 14:36:17,667 DEBUG lava-logs => PING(20) 2019-10-31 14:36:22,490 INFO scheduling health checks: 2019-10-31 14:36:22,494 INFO scheduling jobs: 2019-10-31 14:36:31,386 DEBUG Dhanu => PING(20) 2019-10-31 14:36:37,690 DEBUG lava-logs => PING(20) 2019-10-31 14:36:42,507 INFO scheduling health checks: 2019-10-31 14:36:42,511 INFO scheduling jobs: 2019-10-31 14:36:51,410 DEBUG Dhanu => PING(20) 2019-10-31 14:36:57,710 DEBUG lava-logs => PING(20) 2019-10-31 14:37:02,526 INFO scheduling health checks: 2019-10-31 14:37:02,530 INFO scheduling jobs: 2019-10-31 14:37:11,433 DEBUG Dhanu => PING(20) 2019-10-31 14:37:17,733 DEBUG lava-logs => PING(20) 2019-10-31 14:37:22,545 INFO scheduling health checks: 2019-10-31 14:37:22,548 INFO scheduling jobs: 2019-10-31 14:37:31,457 DEBUG Dhanu => PING(20) 2019-10-31 14:37:37,756 DEBUG lava-logs => PING(20) 2019-10-31 14:37:42,562 INFO scheduling health checks: 2019-10-31 14:37:42,566 INFO scheduling jobs: 2019-10-31 14:37:51,482 DEBUG Dhanu => PING(20) 2019-10-31 14:37:57,779 DEBUG lava-logs => PING(20) 2019-10-31 14:38:02,582 INFO scheduling health checks: 2019-10-31 14:38:02,586 INFO scheduling jobs: 2019-10-31 14:38:11,504 DEBUG Dhanu => PING(20) 2019-10-31 14:38:17,802 DEBUG lava-logs => PING(20) 2019-10-31 14:38:22,603 INFO scheduling health checks: 2019-10-31 14:38:22,607 INFO scheduling jobs: 2019-10-31 14:38:31,528 DEBUG Dhanu => PING(20) 2019-10-31 14:38:37,825 DEBUG lava-logs => PING(20) 2019-10-31 14:38:42,624 INFO scheduling health checks: 2019-10-31 14:38:42,628 INFO scheduling jobs: 2019-10-31 14:38:51,550 DEBUG Dhanu => PING(20) 2019-10-31 14:38:57,839 DEBUG lava-logs => PING(20) 2019-10-31 14:39:02,643 INFO scheduling health checks: 2019-10-31 14:39:02,647 INFO scheduling jobs: 2019-10-31 14:39:11,574 DEBUG Dhanu => PING(20) 2019-10-31 14:39:17,862 DEBUG lava-logs => PING(20) 2019-10-31 14:39:22,659 INFO scheduling health checks: 2019-10-31 14:39:22,663 INFO scheduling jobs: 2019-10-31 14:39:31,595 DEBUG Dhanu => PING(20) 2019-10-31 14:39:37,885 DEBUG lava-logs => PING(20) 2019-10-31 14:39:42,678 INFO scheduling health checks: 2019-10-31 14:39:42,680 INFO scheduling jobs: 2019-10-31 14:39:51,619 DEBUG Dhanu => PING(20) 2019-10-31 14:39:57,903 DEBUG lava-logs => PING(20) 2019-10-31 14:40:02,696 INFO scheduling health checks: 2019-10-31 14:40:02,700 INFO scheduling jobs: 2019-10-31 14:40:11,642 DEBUG Dhanu => PING(20) 2019-10-31 14:40:17,920 DEBUG lava-logs => PING(20) 2019-10-31 14:40:22,714 INFO scheduling health checks: 2019-10-31 14:40:22,720 INFO scheduling jobs: 2019-10-31 14:40:31,660 DEBUG Dhanu => PING(20) 2019-10-31 14:40:37,942 DEBUG lava-logs => PING(20) 2019-10-31 14:40:42,735 INFO scheduling health checks: 2019-10-31 14:40:42,740 INFO scheduling jobs: 2019-10-31 14:40:51,680 DEBUG Dhanu => PING(20) 2019-10-31 14:40:57,962 DEBUG lava-logs => PING(20) 2019-10-31 14:41:02,757 INFO scheduling health checks: 2019-10-31 14:41:02,761 INFO scheduling jobs: 2019-10-31 14:41:11,698 DEBUG Dhanu => PING(20) 2019-10-31 14:41:17,983 DEBUG lava-logs => PING(20) 2019-10-31 14:41:22,774 INFO scheduling health checks: 2019-10-31 14:41:22,778 INFO scheduling jobs: 2019-10-31 14:41:31,722 DEBUG Dhanu => PING(20) 2019-10-31 14:41:38,006 DEBUG lava-logs => PING(20) 2019-10-31 14:41:42,793 INFO scheduling health checks: 2019-10-31 14:41:42,797 INFO scheduling jobs: 2019-10-31 14:41:51,747 DEBUG Dhanu => PING(20) 2019-10-31 14:41:58,028 DEBUG lava-logs => PING(20) 2019-10-31 14:42:02,813 INFO scheduling health checks: 2019-10-31 14:42:02,817 INFO scheduling jobs: 2019-10-31 14:42:11,771 DEBUG Dhanu => PING(20) 2019-10-31 14:42:18,046 DEBUG lava-logs => PING(20) 2019-10-31 14:42:22,834 INFO scheduling health checks: 2019-10-31 14:42:22,838 INFO scheduling jobs: 2019-10-31 14:42:31,796 DEBUG Dhanu => PING(20) 2019-10-31 14:42:38,068 DEBUG lava-logs => PING(20) 2019-10-31 14:42:42,859 INFO scheduling health checks: 2019-10-31 14:42:42,863 INFO scheduling jobs: 2019-10-31 14:42:51,814 DEBUG Dhanu => PING(20) 2019-10-31 14:42:58,091 DEBUG lava-logs => PING(20) 2019-10-31 14:43:02,878 INFO scheduling health checks: 2019-10-31 14:43:02,882 INFO scheduling jobs: 2019-10-31 14:43:11,832 DEBUG Dhanu => PING(20) 2019-10-31 14:43:18,114 DEBUG lava-logs => PING(20) 2019-10-31 14:43:22,897 INFO scheduling health checks: 2019-10-31 14:43:22,901 INFO scheduling jobs: 2019-10-31 14:43:31,852 DEBUG Dhanu => PING(20) 2019-10-31 14:43:33,768 DEBUG lava-logs => PING(20) 2019-10-31 14:43:33,794 DEBUG lava-logs => PING(20) 2019-10-31 14:43:42,915 INFO scheduling health checks: 2019-10-31 14:43:42,921 INFO scheduling jobs: 2019-10-31 14:43:51,870 DEBUG Dhanu => PING(20) 2019-10-31 14:43:53,812 DEBUG lava-logs => PING(20) 2019-10-31 14:44:02,934 INFO scheduling health checks: 2019-10-31 14:44:02,936 INFO scheduling jobs: 2019-10-31 14:44:12,320 DEBUG Dhanu => PING(20) 2019-10-31 14:44:13,872 DEBUG lava-logs => PING(20) 2019-10-31 14:44:22,951 INFO scheduling health checks: 2019-10-31 14:44:22,955 INFO scheduling jobs: 2019-10-31 14:44:32,344 DEBUG Dhanu => PING(20) 2019-10-31 14:44:33,896 DEBUG lava-logs => PING(20) 2019-10-31 14:44:43,019 INFO scheduling health checks: 2019-10-31 14:44:43,024 INFO scheduling jobs: 2019-10-31 14:44:52,368 DEBUG Dhanu => PING(20) 2019-10-31 14:44:53,918 DEBUG lava-logs => PING(20) 2019-10-31 14:45:03,039 INFO scheduling health checks: 2019-10-31 14:45:03,043 INFO scheduling jobs: 2019-10-31 14:45:12,392 DEBUG Dhanu => PING(20) 2019-10-31 14:45:13,939 DEBUG lava-logs => PING(20) 2019-10-31 14:45:23,057 INFO scheduling health checks: 2019-10-31 14:45:23,061 INFO scheduling jobs: 2019-10-31 14:45:32,411 DEBUG Dhanu => PING(20) 2019-10-31 14:45:33,952 DEBUG lava-logs => PING(20) 2019-10-31 14:45:43,077 INFO scheduling health checks: 2019-10-31 14:45:43,081 INFO scheduling jobs: 2019-10-31 14:45:52,436 DEBUG Dhanu => PING(20) 2019-10-31 14:45:53,975 DEBUG lava-logs => PING(20) 2019-10-31 14:46:03,099 INFO scheduling health checks: 2019-10-31 14:46:03,101 INFO scheduling jobs: 2019-10-31 14:46:12,459 DEBUG Dhanu => PING(20) 2019-10-31 14:46:14,025 DEBUG lava-logs => PING(20) 2019-10-31 14:46:23,111 INFO scheduling health checks: 2019-10-31 14:46:23,113 INFO scheduling jobs: 2019-10-31 14:46:32,478 DEBUG Dhanu => PING(20) 2019-10-31 14:46:34,047 DEBUG lava-logs => PING(20) 2019-10-31 14:46:43,122 INFO scheduling health checks: 2019-10-31 14:46:43,126 INFO scheduling jobs: 2019-10-31 14:46:52,500 DEBUG Dhanu => PING(20) 2019-10-31 14:46:54,070 DEBUG lava-logs => PING(20) 2019-10-31 14:47:03,146 INFO scheduling health checks: 2019-10-31 14:47:03,149 INFO scheduling jobs: 2019-10-31 14:47:12,519 DEBUG Dhanu => PING(20) 2019-10-31 14:47:14,093 DEBUG lava-logs => PING(20) 2019-10-31 14:47:23,167 INFO scheduling health checks: 2019-10-31 14:47:23,170 INFO scheduling jobs: 2019-10-31 14:47:32,539 DEBUG Dhanu => PING(20) 2019-10-31 14:47:34,115 DEBUG lava-logs => PING(20) 2019-10-31 14:47:43,188 INFO scheduling health checks: 2019-10-31 14:47:43,192 INFO scheduling jobs: 2019-10-31 14:47:52,560 DEBUG Dhanu => PING(20) 2019-10-31 14:47:54,138 DEBUG lava-logs => PING(20) 2019-10-31 14:48:03,210 INFO scheduling health checks: 2019-10-31 14:48:03,213 INFO scheduling jobs: 2019-10-31 14:48:12,578 DEBUG Dhanu => PING(20) 2019-10-31 14:48:14,159 DEBUG lava-logs => PING(20) 2019-10-31 14:48:23,228 INFO scheduling health checks: 2019-10-31 14:48:23,232 INFO scheduling jobs: 2019-10-31 14:48:32,600 DEBUG Dhanu => PING(20) 2019-10-31 14:48:34,180 DEBUG lava-logs => PING(20) 2019-10-31 14:48:43,250 INFO scheduling health checks: 2019-10-31 14:48:43,254 INFO scheduling jobs: 2019-10-31 14:48:52,618 DEBUG Dhanu => PING(20) 2019-10-31 14:48:54,201 DEBUG lava-logs => PING(20) 2019-10-31 14:49:03,272 INFO scheduling health checks: 2019-10-31 14:49:03,276 INFO scheduling jobs: 2019-10-31 14:49:12,641 DEBUG Dhanu => PING(20) 2019-10-31 14:49:14,224 DEBUG lava-logs => PING(20) 2019-10-31 14:49:23,294 INFO scheduling health checks: 2019-10-31 14:49:23,298 INFO scheduling jobs: 2019-10-31 14:49:32,664 DEBUG Dhanu => PING(20) 2019-10-31 14:49:34,243 DEBUG lava-logs => PING(20) 2019-10-31 14:49:43,314 INFO scheduling health checks: 2019-10-31 14:49:43,317 INFO scheduling jobs: 2019-10-31 14:49:52,764 DEBUG Dhanu => PING(20) 2019-10-31 14:49:54,266 DEBUG lava-logs => PING(20) 2019-10-31 14:50:03,335 INFO scheduling health checks: 2019-10-31 14:50:03,338 INFO scheduling jobs: 2019-10-31 14:50:12,882 DEBUG Dhanu => PING(20) 2019-10-31 14:50:13,760 INFO [POLL] Received a signal, leaving 2019-10-31 14:50:13,761 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 14:50:30,127 INFO [INIT] Dropping privileges 2019-10-31 14:50:30,127 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 14:50:30,128 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 14:50:30,129 INFO [INIT] Marking all workers as offline 2019-10-31 14:50:30,169 INFO [INIT] LAVA master has started. 2019-10-31 14:50:30,169 INFO [INIT] Using protocol version 3 2019-10-31 14:50:30,219 DEBUG lava-logs => PING(20) 2019-10-31 14:50:32,597 INFO Dhanu => HELLO 2019-10-31 14:50:32,598 WARNING New dispatcher 2019-10-31 14:50:37,604 DEBUG Dhanu => PING(20) 2019-10-31 14:50:40,228 DEBUG lava-logs => PING(20) 2019-10-31 14:50:50,171 INFO scheduling health checks: 2019-10-31 14:50:50,178 INFO scheduling jobs: 2019-10-31 14:50:57,635 DEBUG Dhanu => PING(20) 2019-10-31 14:51:05,588 DEBUG lava-logs => PING(20) 2019-10-31 14:51:05,615 DEBUG lava-logs => PING(20) 2019-10-31 14:51:10,193 INFO scheduling health checks: 2019-10-31 14:51:10,196 INFO scheduling jobs: 2019-10-31 14:51:17,652 DEBUG Dhanu => PING(20) 2019-10-31 14:51:25,638 DEBUG lava-logs => PING(20) 2019-10-31 14:51:30,214 INFO scheduling health checks: 2019-10-31 14:51:30,220 INFO scheduling jobs: 2019-10-31 14:51:37,668 DEBUG Dhanu => PING(20) 2019-10-31 14:51:45,655 DEBUG lava-logs => PING(20) 2019-10-31 14:51:50,237 INFO scheduling health checks: 2019-10-31 14:51:50,240 INFO scheduling jobs: 2019-10-31 14:51:57,684 DEBUG Dhanu => PING(20) 2019-10-31 14:52:05,678 DEBUG lava-logs => PING(20) 2019-10-31 14:52:10,255 INFO scheduling health checks: 2019-10-31 14:52:10,260 INFO scheduling jobs: 2019-10-31 14:52:17,701 DEBUG Dhanu => PING(20) 2019-10-31 14:52:25,692 DEBUG lava-logs => PING(20) 2019-10-31 14:52:30,275 INFO scheduling health checks: 2019-10-31 14:52:30,279 INFO scheduling jobs: 2019-10-31 14:52:37,721 DEBUG Dhanu => PING(20) 2019-10-31 14:52:45,715 DEBUG lava-logs => PING(20) 2019-10-31 14:52:50,296 INFO scheduling health checks: 2019-10-31 14:52:50,298 INFO scheduling jobs: 2019-10-31 14:52:57,743 DEBUG Dhanu => PING(20) 2019-10-31 14:53:05,732 DEBUG lava-logs => PING(20) 2019-10-31 14:53:10,306 INFO scheduling health checks: 2019-10-31 14:53:10,310 INFO scheduling jobs: 2019-10-31 14:53:17,759 DEBUG Dhanu => PING(20) 2019-10-31 14:53:25,768 DEBUG lava-logs => PING(20) 2019-10-31 14:53:30,330 INFO scheduling health checks: 2019-10-31 14:53:30,344 INFO scheduling jobs: 2019-10-31 14:53:37,776 DEBUG Dhanu => PING(20) 2019-10-31 14:53:45,790 DEBUG lava-logs => PING(20) 2019-10-31 14:53:50,361 INFO scheduling health checks: 2019-10-31 14:53:50,370 INFO scheduling jobs: 2019-10-31 14:53:57,800 DEBUG Dhanu => PING(20) 2019-10-31 14:54:05,810 DEBUG lava-logs => PING(20) 2019-10-31 14:54:10,383 INFO scheduling health checks: 2019-10-31 14:54:10,394 INFO scheduling jobs: 2019-10-31 14:54:17,820 DEBUG Dhanu => PING(20) 2019-10-31 14:54:25,831 DEBUG lava-logs => PING(20) 2019-10-31 14:54:30,407 INFO scheduling health checks: 2019-10-31 14:54:30,420 INFO scheduling jobs: 2019-10-31 14:54:37,839 DEBUG Dhanu => PING(20) 2019-10-31 14:54:45,854 DEBUG lava-logs => PING(20) 2019-10-31 14:54:50,433 INFO scheduling health checks: 2019-10-31 14:54:50,444 INFO scheduling jobs: 2019-10-31 14:54:57,860 DEBUG Dhanu => PING(20) 2019-10-31 14:55:05,874 DEBUG lava-logs => PING(20) 2019-10-31 14:55:10,461 INFO scheduling health checks: 2019-10-31 14:55:10,471 INFO scheduling jobs: 2019-10-31 14:55:17,878 DEBUG Dhanu => PING(20) 2019-10-31 14:55:25,892 DEBUG lava-logs => PING(20) 2019-10-31 14:55:30,485 INFO scheduling health checks: 2019-10-31 14:55:30,493 INFO scheduling jobs: 2019-10-31 14:55:37,899 DEBUG Dhanu => PING(20) 2019-10-31 14:55:45,905 DEBUG lava-logs => PING(20) 2019-10-31 14:55:50,509 INFO scheduling health checks: 2019-10-31 14:55:50,513 INFO scheduling jobs: 2019-10-31 14:55:57,920 DEBUG Dhanu => PING(20) 2019-10-31 14:56:06,025 DEBUG lava-logs => PING(20) 2019-10-31 14:56:10,533 INFO scheduling health checks: 2019-10-31 14:56:10,537 INFO scheduling jobs: 2019-10-31 14:56:17,956 DEBUG Dhanu => PING(20) 2019-10-31 14:56:26,046 DEBUG lava-logs => PING(20) 2019-10-31 14:56:30,553 INFO scheduling health checks: 2019-10-31 14:56:30,563 INFO scheduling jobs: 2019-10-31 14:56:37,972 DEBUG Dhanu => PING(20) 2019-10-31 14:56:46,070 DEBUG lava-logs => PING(20) 2019-10-31 14:56:50,574 INFO scheduling health checks: 2019-10-31 14:56:50,582 INFO scheduling jobs: 2019-10-31 14:56:57,994 DEBUG Dhanu => PING(20) 2019-10-31 14:57:06,116 DEBUG lava-logs => PING(20) 2019-10-31 14:57:10,595 INFO scheduling health checks: 2019-10-31 14:57:10,618 INFO scheduling jobs: 2019-10-31 14:57:18,052 DEBUG Dhanu => PING(20) 2019-10-31 14:57:26,136 DEBUG lava-logs => PING(20) 2019-10-31 14:57:30,631 INFO scheduling health checks: 2019-10-31 14:57:30,635 INFO scheduling jobs: 2019-10-31 14:57:38,068 DEBUG Dhanu => PING(20) 2019-10-31 14:57:46,158 DEBUG lava-logs => PING(20) 2019-10-31 14:57:50,653 INFO scheduling health checks: 2019-10-31 14:57:50,654 INFO scheduling jobs: 2019-10-31 14:57:58,088 DEBUG Dhanu => PING(20) 2019-10-31 14:58:06,181 DEBUG lava-logs => PING(20) 2019-10-31 14:58:10,663 INFO scheduling health checks: 2019-10-31 14:58:10,667 INFO scheduling jobs: 2019-10-31 14:58:18,123 DEBUG Dhanu => PING(20) 2019-10-31 14:58:26,204 DEBUG lava-logs => PING(20) 2019-10-31 14:58:30,681 INFO scheduling health checks: 2019-10-31 14:58:30,685 INFO scheduling jobs: 2019-10-31 14:58:38,147 DEBUG Dhanu => PING(20) 2019-10-31 14:58:46,223 DEBUG lava-logs => PING(20) 2019-10-31 14:58:50,700 INFO scheduling health checks: 2019-10-31 14:58:50,704 INFO scheduling jobs: 2019-10-31 14:58:58,168 DEBUG Dhanu => PING(20) 2019-10-31 14:59:06,244 DEBUG lava-logs => PING(20) 2019-10-31 14:59:10,719 INFO scheduling health checks: 2019-10-31 14:59:10,723 INFO scheduling jobs: 2019-10-31 14:59:18,192 DEBUG Dhanu => PING(20) 2019-10-31 14:59:26,264 DEBUG lava-logs => PING(20) 2019-10-31 14:59:30,744 INFO scheduling health checks: 2019-10-31 14:59:30,747 INFO scheduling jobs: 2019-10-31 14:59:38,215 DEBUG Dhanu => PING(20) 2019-10-31 14:59:46,286 DEBUG lava-logs => PING(20) 2019-10-31 14:59:50,761 INFO scheduling health checks: 2019-10-31 14:59:50,765 INFO scheduling jobs: 2019-10-31 14:59:58,236 DEBUG Dhanu => PING(20) 2019-10-31 15:00:06,304 DEBUG lava-logs => PING(20) 2019-10-31 15:00:10,778 INFO scheduling health checks: 2019-10-31 15:00:10,782 INFO scheduling jobs: 2019-10-31 15:00:18,256 DEBUG Dhanu => PING(20) 2019-10-31 15:00:26,324 DEBUG lava-logs => PING(20) 2019-10-31 15:00:30,796 INFO scheduling health checks: 2019-10-31 15:00:30,800 INFO scheduling jobs: 2019-10-31 15:00:38,281 DEBUG Dhanu => PING(20) 2019-10-31 15:00:46,346 DEBUG lava-logs => PING(20) 2019-10-31 15:00:50,812 INFO scheduling health checks: 2019-10-31 15:00:50,816 INFO scheduling jobs: 2019-10-31 15:00:58,312 DEBUG Dhanu => PING(20) 2019-10-31 15:01:06,367 DEBUG lava-logs => PING(20) 2019-10-31 15:01:10,829 INFO scheduling health checks: 2019-10-31 15:01:10,833 INFO scheduling jobs: 2019-10-31 15:01:18,336 DEBUG Dhanu => PING(20) 2019-10-31 15:01:26,390 DEBUG lava-logs => PING(20) 2019-10-31 15:01:30,847 INFO scheduling health checks: 2019-10-31 15:01:30,850 INFO scheduling jobs: 2019-10-31 15:01:38,357 DEBUG Dhanu => PING(20) 2019-10-31 15:01:46,411 DEBUG lava-logs => PING(20) 2019-10-31 15:01:50,868 INFO scheduling health checks: 2019-10-31 15:01:50,871 INFO scheduling jobs: 2019-10-31 15:01:58,381 DEBUG Dhanu => PING(20) 2019-10-31 15:02:06,434 DEBUG lava-logs => PING(20) 2019-10-31 15:02:10,887 INFO scheduling health checks: 2019-10-31 15:02:10,891 INFO scheduling jobs: 2019-10-31 15:02:18,405 DEBUG Dhanu => PING(20) 2019-10-31 15:02:26,457 DEBUG lava-logs => PING(20) 2019-10-31 15:02:30,906 INFO scheduling health checks: 2019-10-31 15:02:30,910 INFO scheduling jobs: 2019-10-31 15:02:38,429 DEBUG Dhanu => PING(20) 2019-10-31 15:02:46,480 DEBUG lava-logs => PING(20) 2019-10-31 15:02:50,925 INFO scheduling health checks: 2019-10-31 15:02:50,929 INFO scheduling jobs: 2019-10-31 15:02:58,448 DEBUG Dhanu => PING(20) 2019-10-31 15:03:06,500 DEBUG lava-logs => PING(20) 2019-10-31 15:03:10,947 INFO scheduling health checks: 2019-10-31 15:03:10,951 INFO scheduling jobs: 2019-10-31 15:03:18,468 DEBUG Dhanu => PING(20) 2019-10-31 15:03:26,520 DEBUG lava-logs => PING(20) 2019-10-31 15:03:30,963 INFO scheduling health checks: 2019-10-31 15:03:30,966 INFO scheduling jobs: 2019-10-31 15:03:38,490 DEBUG Dhanu => PING(20) 2019-10-31 15:03:46,542 DEBUG lava-logs => PING(20) 2019-10-31 15:03:50,981 INFO scheduling health checks: 2019-10-31 15:03:50,984 INFO scheduling jobs: 2019-10-31 15:03:58,511 DEBUG Dhanu => PING(20) 2019-10-31 15:04:06,562 DEBUG lava-logs => PING(20) 2019-10-31 15:04:10,999 INFO scheduling health checks: 2019-10-31 15:04:11,003 INFO scheduling jobs: 2019-10-31 15:04:18,528 DEBUG Dhanu => PING(20) 2019-10-31 15:04:26,578 DEBUG lava-logs => PING(20) 2019-10-31 15:04:31,018 INFO scheduling health checks: 2019-10-31 15:04:31,021 INFO scheduling jobs: 2019-10-31 15:04:38,546 DEBUG Dhanu => PING(20) 2019-10-31 15:04:46,598 DEBUG lava-logs => PING(20) 2019-10-31 15:04:51,036 INFO scheduling health checks: 2019-10-31 15:04:51,039 INFO scheduling jobs: 2019-10-31 15:04:58,564 DEBUG Dhanu => PING(20) 2019-10-31 15:05:06,612 DEBUG lava-logs => PING(20) 2019-10-31 15:05:11,051 INFO scheduling health checks: 2019-10-31 15:05:11,055 INFO scheduling jobs: 2019-10-31 15:05:18,582 DEBUG Dhanu => PING(20) 2019-10-31 15:05:26,632 DEBUG lava-logs => PING(20) 2019-10-31 15:05:31,069 INFO scheduling health checks: 2019-10-31 15:05:31,071 INFO scheduling jobs: 2019-10-31 15:05:38,600 DEBUG Dhanu => PING(20) 2019-10-31 15:05:46,652 DEBUG lava-logs => PING(20) 2019-10-31 15:05:51,082 INFO scheduling health checks: 2019-10-31 15:05:51,086 INFO scheduling jobs: 2019-10-31 15:05:58,616 DEBUG Dhanu => PING(20) 2019-10-31 15:06:06,674 DEBUG lava-logs => PING(20) 2019-10-31 15:06:11,099 INFO scheduling health checks: 2019-10-31 15:06:11,101 INFO scheduling jobs: 2019-10-31 15:06:18,637 DEBUG Dhanu => PING(20) 2019-10-31 15:06:26,696 DEBUG lava-logs => PING(20) 2019-10-31 15:06:31,115 INFO scheduling health checks: 2019-10-31 15:06:31,119 INFO scheduling jobs: 2019-10-31 15:06:38,656 DEBUG Dhanu => PING(20) 2019-10-31 15:06:46,719 DEBUG lava-logs => PING(20) 2019-10-31 15:06:51,137 INFO scheduling health checks: 2019-10-31 15:06:51,141 INFO scheduling jobs: 2019-10-31 15:06:58,677 DEBUG Dhanu => PING(20) 2019-10-31 15:07:06,742 DEBUG lava-logs => PING(20) 2019-10-31 15:07:11,155 INFO scheduling health checks: 2019-10-31 15:07:11,159 INFO scheduling jobs: 2019-10-31 15:07:18,702 DEBUG Dhanu => PING(20) 2019-10-31 15:07:26,765 DEBUG lava-logs => PING(20) 2019-10-31 15:07:31,173 INFO scheduling health checks: 2019-10-31 15:07:31,177 INFO scheduling jobs: 2019-10-31 15:07:38,725 DEBUG Dhanu => PING(20) 2019-10-31 15:07:44,435 DEBUG lava-logs => PING(20) 2019-10-31 15:07:44,458 DEBUG lava-logs => PING(20) 2019-10-31 15:07:51,193 INFO scheduling health checks: 2019-10-31 15:07:51,197 INFO scheduling jobs: 2019-10-31 15:07:58,749 DEBUG Dhanu => PING(20) 2019-10-31 15:08:04,476 DEBUG lava-logs => PING(20) 2019-10-31 15:08:11,215 INFO scheduling health checks: 2019-10-31 15:08:11,219 INFO scheduling jobs: 2019-10-31 15:08:24,499 DEBUG lava-logs => PING(20) 2019-10-31 15:08:31,235 INFO scheduling health checks: 2019-10-31 15:08:31,239 INFO scheduling jobs: 2019-10-31 15:08:44,520 DEBUG lava-logs => PING(20) 2019-10-31 15:08:51,253 INFO scheduling health checks: 2019-10-31 15:08:51,257 INFO scheduling jobs: 2019-10-31 15:09:04,540 DEBUG lava-logs => PING(20) 2019-10-31 15:09:10,351 ERROR [STATE] Dispatcher goes OFFLINE 2019-10-31 15:09:11,273 INFO scheduling health checks: 2019-10-31 15:09:11,276 INFO scheduling jobs: 2019-10-31 15:09:24,556 DEBUG lava-logs => PING(20) 2019-10-31 15:09:31,293 INFO scheduling health checks: 2019-10-31 15:09:31,297 INFO scheduling jobs: 2019-10-31 15:09:44,579 DEBUG lava-logs => PING(20) 2019-10-31 15:09:50,380 INFO [POLL] Received a signal, leaving 2019-10-31 15:09:50,389 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 15:09:51,832 INFO [INIT] Dropping privileges 2019-10-31 15:09:51,832 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 15:09:51,833 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 15:09:51,834 INFO [INIT] Marking all workers as offline 2019-10-31 15:09:51,894 INFO [INIT] Starting encryption 2019-10-31 15:09:51,896 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-10-31 15:09:51,897 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-10-31 15:09:51,897 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-10-31 15:09:51,917 INFO [INIT] LAVA master has started. 2019-10-31 15:09:51,917 INFO [INIT] Using protocol version 3 2019-10-31 15:09:58,079 INFO Dhanu => HELLO 2019-10-31 15:09:58,080 WARNING New dispatcher 2019-10-31 15:10:03,087 DEBUG Dhanu => PING(20) 2019-10-31 15:10:11,923 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:10:23,104 DEBUG Dhanu => PING(20) 2019-10-31 15:10:31,932 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:10:43,120 DEBUG Dhanu => PING(20) 2019-10-31 15:10:51,939 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:11:03,136 DEBUG Dhanu => PING(20) 2019-10-31 15:11:11,948 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:11:23,160 DEBUG Dhanu => PING(20) 2019-10-31 15:11:31,955 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:11:43,182 DEBUG Dhanu => PING(20) 2019-10-31 15:11:51,963 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:12:03,202 DEBUG Dhanu => PING(20) 2019-10-31 15:12:11,971 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:12:23,220 DEBUG Dhanu => PING(20) 2019-10-31 15:12:32,005 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:12:43,396 DEBUG Dhanu => PING(20) 2019-10-31 15:12:52,015 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:13:03,511 DEBUG Dhanu => PING(20) 2019-10-31 15:13:12,021 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:13:23,532 DEBUG Dhanu => PING(20) 2019-10-31 15:13:32,029 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:13:43,552 DEBUG Dhanu => PING(20) 2019-10-31 15:13:52,037 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:14:03,577 DEBUG Dhanu => PING(20) 2019-10-31 15:14:12,043 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:14:23,600 DEBUG Dhanu => PING(20) 2019-10-31 15:14:32,050 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:14:43,618 DEBUG Dhanu => PING(20) 2019-10-31 15:14:52,057 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:15:03,636 DEBUG Dhanu => PING(20) 2019-10-31 15:15:12,064 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:15:23,652 DEBUG Dhanu => PING(20) 2019-10-31 15:15:32,068 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:15:43,672 DEBUG Dhanu => PING(20) 2019-10-31 15:15:52,074 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:16:03,688 DEBUG Dhanu => PING(20) 2019-10-31 15:16:12,081 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:16:23,705 DEBUG Dhanu => PING(20) 2019-10-31 15:16:32,088 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:16:43,728 DEBUG Dhanu => PING(20) 2019-10-31 15:16:52,096 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:17:03,744 DEBUG Dhanu => PING(20) 2019-10-31 15:17:12,105 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:17:23,764 DEBUG Dhanu => PING(20) 2019-10-31 15:17:32,113 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:17:43,786 DEBUG Dhanu => PING(20) 2019-10-31 15:17:52,121 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:18:03,808 DEBUG Dhanu => PING(20) 2019-10-31 15:18:12,128 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:18:23,824 DEBUG Dhanu => PING(20) 2019-10-31 15:18:32,136 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:18:43,848 DEBUG Dhanu => PING(20) 2019-10-31 15:18:52,142 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:19:03,868 DEBUG Dhanu => PING(20) 2019-10-31 15:19:12,150 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:19:23,892 DEBUG Dhanu => PING(20) 2019-10-31 15:19:32,158 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:19:43,912 DEBUG Dhanu => PING(20) 2019-10-31 15:19:52,166 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:20:03,928 DEBUG Dhanu => PING(20) 2019-10-31 15:20:12,173 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:20:23,944 DEBUG Dhanu => PING(20) 2019-10-31 15:20:32,178 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:20:43,963 DEBUG Dhanu => PING(20) 2019-10-31 15:20:52,185 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 15:21:03,980 DEBUG Dhanu => PING(20) 2019-10-31 15:21:12,193 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:57:18,417 INFO [POLL] Received a signal, leaving 2019-10-31 09:57:18,418 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 09:57:23,503 INFO [INIT] Dropping privileges 2019-10-31 09:57:23,503 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 09:57:23,503 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 09:57:23,505 INFO [INIT] Marking all workers as offline 2019-10-31 09:57:23,528 INFO [INIT] Starting encryption 2019-10-31 09:57:23,530 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-10-31 09:57:23,530 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-10-31 09:57:23,530 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-10-31 09:57:23,546 INFO [INIT] LAVA master has started. 2019-10-31 09:57:23,546 INFO [INIT] Using protocol version 3 2019-10-31 09:57:27,152 INFO Dhanu => HELLO 2019-10-31 09:57:27,153 WARNING New dispatcher 2019-10-31 09:57:32,156 DEBUG Dhanu => PING(20) 2019-10-31 09:57:43,557 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:57:52,188 DEBUG Dhanu => PING(20) 2019-10-31 09:58:03,567 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:58:12,204 DEBUG Dhanu => PING(20) 2019-10-31 09:58:23,573 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:58:32,228 DEBUG Dhanu => PING(20) 2019-10-31 09:58:43,581 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:58:52,245 DEBUG Dhanu => PING(20) 2019-10-31 09:59:03,591 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:59:12,384 DEBUG Dhanu => PING(20) 2019-10-31 09:59:23,619 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:59:32,405 DEBUG Dhanu => PING(20) 2019-10-31 09:59:43,628 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 09:59:52,430 DEBUG Dhanu => PING(20) 2019-10-31 10:00:03,636 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:00:12,448 DEBUG Dhanu => PING(20) 2019-10-31 10:00:23,645 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:00:32,473 DEBUG Dhanu => PING(20) 2019-10-31 10:00:43,652 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:00:52,492 DEBUG Dhanu => PING(20) 2019-10-31 10:01:03,658 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:01:12,512 DEBUG Dhanu => PING(20) 2019-10-31 10:01:23,667 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:01:32,537 DEBUG Dhanu => PING(20) 2019-10-31 10:01:43,675 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:01:52,555 DEBUG Dhanu => PING(20) 2019-10-31 10:02:03,683 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:02:12,575 DEBUG Dhanu => PING(20) 2019-10-31 10:02:23,692 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:02:32,614 DEBUG Dhanu => PING(20) 2019-10-31 10:02:43,697 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:02:52,633 DEBUG Dhanu => PING(20) 2019-10-31 10:03:03,704 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:03:12,651 DEBUG Dhanu => PING(20) 2019-10-31 10:03:23,711 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:03:32,668 DEBUG Dhanu => PING(20) 2019-10-31 10:03:43,721 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:03:52,684 DEBUG Dhanu => PING(20) 2019-10-31 10:04:03,728 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:04:12,707 DEBUG Dhanu => PING(20) 2019-10-31 10:04:23,736 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:04:32,727 DEBUG Dhanu => PING(20) 2019-10-31 10:04:43,744 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:04:52,745 DEBUG Dhanu => PING(20) 2019-10-31 10:05:03,750 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:05:12,770 DEBUG Dhanu => PING(20) 2019-10-31 10:05:23,758 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:05:32,795 DEBUG Dhanu => PING(20) 2019-10-31 10:05:43,765 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:05:52,819 DEBUG Dhanu => PING(20) 2019-10-31 10:06:03,773 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:06:12,840 DEBUG Dhanu => PING(20) 2019-10-31 10:06:23,781 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:06:32,857 DEBUG Dhanu => PING(20) 2019-10-31 10:06:43,789 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:06:52,875 DEBUG Dhanu => PING(20) 2019-10-31 10:07:03,794 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:07:12,895 DEBUG Dhanu => PING(20) 2019-10-31 10:07:23,805 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:07:32,919 DEBUG Dhanu => PING(20) 2019-10-31 10:07:43,815 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:07:52,937 DEBUG Dhanu => PING(20) 2019-10-31 10:08:03,826 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:08:12,962 DEBUG Dhanu => PING(20) 2019-10-31 10:08:23,836 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:08:32,985 DEBUG Dhanu => PING(20) 2019-10-31 10:08:43,847 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:08:53,004 DEBUG Dhanu => PING(20) 2019-10-31 10:09:03,954 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:09:13,025 DEBUG Dhanu => PING(20) 2019-10-31 10:09:23,962 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:09:33,044 DEBUG Dhanu => PING(20) 2019-10-31 10:09:43,971 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:09:53,063 DEBUG Dhanu => PING(20) 2019-10-31 10:10:03,978 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:10:13,081 DEBUG Dhanu => PING(20) 2019-10-31 10:10:23,985 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:10:33,101 DEBUG Dhanu => PING(20) 2019-10-31 10:10:43,994 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:10:53,120 DEBUG Dhanu => PING(20) 2019-10-31 10:11:04,000 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:11:13,143 DEBUG Dhanu => PING(20) 2019-10-31 10:11:24,008 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:11:33,166 DEBUG Dhanu => PING(20) 2019-10-31 10:11:44,016 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:11:53,189 DEBUG Dhanu => PING(20) 2019-10-31 10:12:04,021 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:12:13,211 DEBUG Dhanu => PING(20) 2019-10-31 10:12:24,028 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:12:33,235 DEBUG Dhanu => PING(20) 2019-10-31 10:12:44,037 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:12:53,259 DEBUG Dhanu => PING(20) 2019-10-31 10:13:04,044 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:13:13,280 DEBUG Dhanu => PING(20) 2019-10-31 10:13:24,053 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:13:33,297 DEBUG Dhanu => PING(20) 2019-10-31 10:13:44,063 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:13:53,317 DEBUG Dhanu => PING(20) 2019-10-31 10:14:04,071 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:14:13,337 DEBUG Dhanu => PING(20) 2019-10-31 10:14:24,079 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:14:33,360 DEBUG Dhanu => PING(20) 2019-10-31 10:14:44,087 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:14:53,381 DEBUG Dhanu => PING(20) 2019-10-31 10:15:04,097 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:15:13,399 DEBUG Dhanu => PING(20) 2019-10-31 10:15:24,105 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:15:33,421 DEBUG Dhanu => PING(20) 2019-10-31 10:15:44,114 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:15:53,444 DEBUG Dhanu => PING(20) 2019-10-31 10:16:04,122 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:16:13,464 DEBUG Dhanu => PING(20) 2019-10-31 10:16:24,130 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:16:33,484 DEBUG Dhanu => PING(20) 2019-10-31 10:16:44,137 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:16:53,500 DEBUG Dhanu => PING(20) 2019-10-31 10:17:04,145 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:17:13,519 DEBUG Dhanu => PING(20) 2019-10-31 10:17:24,153 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:17:33,540 DEBUG Dhanu => PING(20) 2019-10-31 10:17:44,159 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:17:53,556 DEBUG Dhanu => PING(20) 2019-10-31 10:17:58,366 INFO scheduling health checks: 2019-10-31 10:17:58,725 INFO scheduling jobs: 2019-10-31 10:17:58,727 DEBUG - qemu 2019-10-31 10:17:58,909 ERROR [CLOSE] Unknown exception raised, leaving! 2019-10-31 10:17:58,909 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 799, in main_loop jobs = schedule(self.logger, self.events["available_dt"]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-10-31 10:17:58,951 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 10:18:05,372 INFO [INIT] Dropping privileges 2019-10-31 10:18:05,372 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 10:18:05,372 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 10:18:05,374 INFO [INIT] Marking all workers as offline 2019-10-31 10:18:05,416 INFO [INIT] Starting encryption 2019-10-31 10:18:05,418 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-10-31 10:18:05,419 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-10-31 10:18:05,419 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-10-31 10:18:05,441 INFO [INIT] LAVA master has started. 2019-10-31 10:18:05,441 INFO [INIT] Using protocol version 3 2019-10-31 10:18:13,573 DEBUG Dhanu => PING(20) 2019-10-31 10:18:13,575 WARNING Unknown dispatcher (server crashed) 2019-10-31 10:18:25,454 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:18:33,591 DEBUG Dhanu => PING(20) 2019-10-31 10:18:45,486 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:18:53,613 DEBUG Dhanu => PING(20) 2019-10-31 10:19:05,538 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:19:13,633 DEBUG Dhanu => PING(20) 2019-10-31 10:19:25,548 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:19:33,651 DEBUG Dhanu => PING(20) 2019-10-31 10:19:45,556 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:19:53,667 DEBUG Dhanu => PING(20) 2019-10-31 10:20:05,564 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:20:13,683 DEBUG Dhanu => PING(20) 2019-10-31 10:20:25,572 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:20:33,703 DEBUG Dhanu => PING(20) 2019-10-31 10:20:45,579 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:20:53,721 DEBUG Dhanu => PING(20) 2019-10-31 10:21:05,587 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:21:13,746 DEBUG Dhanu => PING(20) 2019-10-31 10:21:25,595 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:21:33,766 DEBUG Dhanu => PING(20) 2019-10-31 10:21:45,603 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:21:53,784 DEBUG Dhanu => PING(20) 2019-10-31 10:22:05,610 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:22:13,799 DEBUG Dhanu => PING(20) 2019-10-31 10:22:25,618 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:22:33,815 DEBUG Dhanu => PING(20) 2019-10-31 10:22:45,628 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:22:53,837 DEBUG Dhanu => PING(20) 2019-10-31 10:23:05,631 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:23:13,860 DEBUG Dhanu => PING(20) 2019-10-31 10:23:25,639 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:23:33,885 DEBUG Dhanu => PING(20) 2019-10-31 10:23:45,647 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:23:53,908 DEBUG Dhanu => PING(20) 2019-10-31 10:24:05,653 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:24:13,933 DEBUG Dhanu => PING(20) 2019-10-31 10:24:25,660 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:24:33,953 DEBUG Dhanu => PING(20) 2019-10-31 10:24:45,671 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:24:53,978 DEBUG Dhanu => PING(20) 2019-10-31 10:25:05,683 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:25:14,000 DEBUG Dhanu => PING(20) 2019-10-31 10:25:25,694 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:25:34,015 DEBUG Dhanu => PING(20) 2019-10-31 10:25:45,705 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:25:54,033 DEBUG Dhanu => PING(20) 2019-10-31 10:26:05,715 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:26:14,052 DEBUG Dhanu => PING(20) 2019-10-31 10:26:25,726 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:26:34,071 DEBUG Dhanu => PING(20) 2019-10-31 10:26:45,737 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:26:54,091 DEBUG Dhanu => PING(20) 2019-10-31 10:27:05,747 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:27:14,107 DEBUG Dhanu => PING(20) 2019-10-31 10:27:25,754 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:27:34,125 DEBUG Dhanu => PING(20) 2019-10-31 10:27:45,766 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:27:54,150 DEBUG Dhanu => PING(20) 2019-10-31 10:28:05,774 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:28:14,171 DEBUG Dhanu => PING(20) 2019-10-31 10:28:25,786 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:28:34,189 DEBUG Dhanu => PING(20) 2019-10-31 10:28:45,797 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:28:54,207 DEBUG Dhanu => PING(20) 2019-10-31 10:29:05,807 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:29:14,226 DEBUG Dhanu => PING(20) 2019-10-31 10:29:25,816 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:29:34,244 DEBUG Dhanu => PING(20) 2019-10-31 10:29:45,824 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:29:54,260 DEBUG Dhanu => PING(20) 2019-10-31 10:30:05,831 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:30:14,279 DEBUG Dhanu => PING(20) 2019-10-31 10:30:25,838 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:30:34,297 DEBUG Dhanu => PING(20) 2019-10-31 10:30:45,850 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:30:54,316 DEBUG Dhanu => PING(20) 2019-10-31 10:31:05,861 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:31:14,336 DEBUG Dhanu => PING(20) 2019-10-31 10:31:25,872 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:31:34,355 DEBUG Dhanu => PING(20) 2019-10-31 10:31:45,882 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:31:54,373 DEBUG Dhanu => PING(20) 2019-10-31 10:32:05,890 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:32:14,391 DEBUG Dhanu => PING(20) 2019-10-31 10:32:25,902 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:32:34,412 DEBUG Dhanu => PING(20) 2019-10-31 10:32:45,909 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:32:54,432 DEBUG Dhanu => PING(20) 2019-10-31 10:33:05,919 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:33:14,453 DEBUG Dhanu => PING(20) 2019-10-31 10:33:25,930 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:33:34,472 DEBUG Dhanu => PING(20) 2019-10-31 10:33:45,938 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:33:54,491 DEBUG Dhanu => PING(20) 2019-10-31 10:34:05,950 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:34:14,515 DEBUG Dhanu => PING(20) 2019-10-31 10:34:25,961 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:34:34,535 DEBUG Dhanu => PING(20) 2019-10-31 10:34:45,972 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:34:54,557 DEBUG Dhanu => PING(20) 2019-10-31 10:35:05,984 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:35:14,582 DEBUG Dhanu => PING(20) 2019-10-31 10:35:25,991 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:35:34,605 DEBUG Dhanu => PING(20) 2019-10-31 10:35:46,002 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:35:54,628 DEBUG Dhanu => PING(20) 2019-10-31 10:36:06,011 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:36:14,652 DEBUG Dhanu => PING(20) 2019-10-31 10:36:26,018 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:36:34,672 DEBUG Dhanu => PING(20) 2019-10-31 10:36:46,025 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:36:54,696 DEBUG Dhanu => PING(20) 2019-10-31 10:37:06,033 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:37:14,720 DEBUG Dhanu => PING(20) 2019-10-31 10:37:26,041 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:37:34,744 DEBUG Dhanu => PING(20) 2019-10-31 10:37:46,048 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:37:54,765 DEBUG Dhanu => PING(20) 2019-10-31 10:38:06,056 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:38:14,790 DEBUG Dhanu => PING(20) 2019-10-31 10:38:26,067 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:38:34,814 DEBUG Dhanu => PING(20) 2019-10-31 10:38:46,078 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:38:54,839 DEBUG Dhanu => PING(20) 2019-10-31 10:39:06,086 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:39:14,863 DEBUG Dhanu => PING(20) 2019-10-31 10:39:26,096 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:39:34,884 DEBUG Dhanu => PING(20) 2019-10-31 10:39:46,107 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:39:54,908 DEBUG Dhanu => PING(20) 2019-10-31 10:40:06,118 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:40:14,933 DEBUG Dhanu => PING(20) 2019-10-31 10:40:26,129 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:40:34,957 DEBUG Dhanu => PING(20) 2019-10-31 10:40:46,140 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:40:54,982 DEBUG Dhanu => PING(20) 2019-10-31 10:41:06,151 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:41:15,004 DEBUG Dhanu => PING(20) 2019-10-31 10:41:26,162 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:41:35,028 DEBUG Dhanu => PING(20) 2019-10-31 10:41:46,169 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:41:55,052 DEBUG Dhanu => PING(20) 2019-10-31 10:42:06,177 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:42:15,076 DEBUG Dhanu => PING(20) 2019-10-31 10:42:26,186 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:42:35,098 DEBUG Dhanu => PING(20) 2019-10-31 10:42:46,197 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:42:55,123 DEBUG Dhanu => PING(20) 2019-10-31 10:43:06,204 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:43:15,144 DEBUG Dhanu => PING(20) 2019-10-31 10:43:26,214 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:43:35,168 DEBUG Dhanu => PING(20) 2019-10-31 10:43:46,225 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:43:55,189 DEBUG Dhanu => PING(20) 2019-10-31 10:44:06,236 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:44:15,214 DEBUG Dhanu => PING(20) 2019-10-31 10:44:26,248 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:44:35,236 DEBUG Dhanu => PING(20) 2019-10-31 10:44:46,259 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:44:55,259 DEBUG Dhanu => PING(20) 2019-10-31 10:45:06,269 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:45:15,283 DEBUG Dhanu => PING(20) 2019-10-31 10:45:26,280 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:45:35,307 DEBUG Dhanu => PING(20) 2019-10-31 10:45:46,291 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:45:55,328 DEBUG Dhanu => PING(20) 2019-10-31 10:46:06,301 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:46:15,349 DEBUG Dhanu => PING(20) 2019-10-31 10:46:26,311 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:46:35,374 DEBUG Dhanu => PING(20) 2019-10-31 10:46:46,318 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:46:55,396 DEBUG Dhanu => PING(20) 2019-10-31 10:47:06,326 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:47:15,421 DEBUG Dhanu => PING(20) 2019-10-31 10:47:26,334 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:47:35,444 DEBUG Dhanu => PING(20) 2019-10-31 10:47:46,342 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:47:55,468 DEBUG Dhanu => PING(20) 2019-10-31 10:48:06,350 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:48:15,492 DEBUG Dhanu => PING(20) 2019-10-31 10:48:26,358 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:48:35,513 DEBUG Dhanu => PING(20) 2019-10-31 10:48:46,366 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:48:55,536 DEBUG Dhanu => PING(20) 2019-10-31 10:49:06,374 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:49:15,549 DEBUG Dhanu => PING(20) 2019-10-31 10:49:26,382 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:49:35,569 DEBUG Dhanu => PING(20) 2019-10-31 10:49:46,392 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:49:55,590 DEBUG Dhanu => PING(20) 2019-10-31 10:50:06,402 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:50:15,613 DEBUG Dhanu => PING(20) 2019-10-31 10:50:26,412 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:50:35,632 DEBUG Dhanu => PING(20) 2019-10-31 10:50:46,423 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:50:55,657 DEBUG Dhanu => PING(20) 2019-10-31 10:51:06,432 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:51:15,680 DEBUG Dhanu => PING(20) 2019-10-31 10:51:26,440 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:51:35,697 DEBUG Dhanu => PING(20) 2019-10-31 10:51:46,449 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:51:55,721 DEBUG Dhanu => PING(20) 2019-10-31 10:52:06,458 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:52:15,744 DEBUG Dhanu => PING(20) 2019-10-31 10:52:26,469 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:52:35,761 DEBUG Dhanu => PING(20) 2019-10-31 10:52:46,478 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:52:55,781 DEBUG Dhanu => PING(20) 2019-10-31 10:53:06,488 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:53:15,799 DEBUG Dhanu => PING(20) 2019-10-31 10:53:26,495 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:53:35,821 DEBUG Dhanu => PING(20) 2019-10-31 10:53:46,504 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:53:55,844 DEBUG Dhanu => PING(20) 2019-10-31 10:54:06,512 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:54:15,865 DEBUG Dhanu => PING(20) 2019-10-31 10:54:26,520 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:54:35,885 DEBUG Dhanu => PING(20) 2019-10-31 10:54:46,529 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:54:55,907 DEBUG Dhanu => PING(20) 2019-10-31 10:55:06,539 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:55:15,925 DEBUG Dhanu => PING(20) 2019-10-31 10:55:26,548 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:55:35,944 DEBUG Dhanu => PING(20) 2019-10-31 10:55:46,559 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:55:55,959 DEBUG Dhanu => PING(20) 2019-10-31 10:56:06,566 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:56:15,983 DEBUG Dhanu => PING(20) 2019-10-31 10:56:26,575 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:56:35,999 DEBUG Dhanu => PING(20) 2019-10-31 10:56:46,584 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:56:56,034 DEBUG Dhanu => PING(20) 2019-10-31 10:57:06,591 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:57:16,052 DEBUG Dhanu => PING(20) 2019-10-31 10:57:26,601 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:57:36,071 DEBUG Dhanu => PING(20) 2019-10-31 10:57:46,609 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:57:56,091 DEBUG Dhanu => PING(20) 2019-10-31 10:58:06,617 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:58:16,111 DEBUG Dhanu => PING(20) 2019-10-31 10:58:26,627 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:58:36,131 DEBUG Dhanu => PING(20) 2019-10-31 10:58:46,638 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:58:56,152 DEBUG Dhanu => PING(20) 2019-10-31 10:59:06,646 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:59:16,178 DEBUG Dhanu => PING(20) 2019-10-31 10:59:26,654 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:59:36,200 DEBUG Dhanu => PING(20) 2019-10-31 10:59:46,663 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 10:59:56,224 DEBUG Dhanu => PING(20) 2019-10-31 11:00:06,674 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:00:16,246 DEBUG Dhanu => PING(20) 2019-10-31 11:00:26,682 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:00:36,267 DEBUG Dhanu => PING(20) 2019-10-31 11:00:46,692 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:00:56,291 DEBUG Dhanu => PING(20) 2019-10-31 11:01:06,700 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:01:16,312 DEBUG Dhanu => PING(20) 2019-10-31 11:01:26,709 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:01:36,331 DEBUG Dhanu => PING(20) 2019-10-31 11:01:46,717 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:01:56,348 DEBUG Dhanu => PING(20) 2019-10-31 11:02:06,727 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:02:16,363 DEBUG Dhanu => PING(20) 2019-10-31 11:02:26,734 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:02:36,387 DEBUG Dhanu => PING(20) 2019-10-31 11:02:46,745 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:02:56,408 DEBUG Dhanu => PING(20) 2019-10-31 11:03:06,755 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:03:16,423 DEBUG Dhanu => PING(20) 2019-10-31 11:03:26,765 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:03:36,438 DEBUG Dhanu => PING(20) 2019-10-31 11:03:46,775 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:03:56,459 DEBUG Dhanu => PING(20) 2019-10-31 11:04:06,783 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:04:16,479 DEBUG Dhanu => PING(20) 2019-10-31 11:04:26,790 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:04:36,501 DEBUG Dhanu => PING(20) 2019-10-31 11:04:46,799 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:04:56,520 DEBUG Dhanu => PING(20) 2019-10-31 11:05:06,806 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:05:16,543 DEBUG Dhanu => PING(20) 2019-10-31 11:05:26,816 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:05:36,566 DEBUG Dhanu => PING(20) 2019-10-31 11:05:46,826 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:05:56,591 DEBUG Dhanu => PING(20) 2019-10-31 11:06:06,834 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:06:16,612 DEBUG Dhanu => PING(20) 2019-10-31 11:06:26,842 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:06:36,631 DEBUG Dhanu => PING(20) 2019-10-31 11:06:46,852 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:06:56,648 DEBUG Dhanu => PING(20) 2019-10-31 11:07:06,860 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:07:16,671 DEBUG Dhanu => PING(20) 2019-10-31 11:07:26,867 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:07:36,692 DEBUG Dhanu => PING(20) 2019-10-31 11:07:46,874 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:07:56,711 DEBUG Dhanu => PING(20) 2019-10-31 11:08:06,883 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:08:16,727 DEBUG Dhanu => PING(20) 2019-10-31 11:08:26,891 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:08:36,743 DEBUG Dhanu => PING(20) 2019-10-31 11:08:46,901 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:08:56,765 DEBUG Dhanu => PING(20) 2019-10-31 11:09:06,910 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:09:16,790 DEBUG Dhanu => PING(20) 2019-10-31 11:09:26,919 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:09:36,812 DEBUG Dhanu => PING(20) 2019-10-31 11:09:46,927 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:09:56,827 DEBUG Dhanu => PING(20) 2019-10-31 11:10:06,934 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:10:16,851 DEBUG Dhanu => PING(20) 2019-10-31 11:10:26,940 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:10:36,867 DEBUG Dhanu => PING(20) 2019-10-31 11:10:46,948 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:10:56,891 DEBUG Dhanu => PING(20) 2019-10-31 11:11:06,956 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:11:16,915 DEBUG Dhanu => PING(20) 2019-10-31 11:11:26,964 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:11:36,935 DEBUG Dhanu => PING(20) 2019-10-31 11:11:46,971 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:11:56,959 DEBUG Dhanu => PING(20) 2019-10-31 11:12:06,978 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:12:16,982 DEBUG Dhanu => PING(20) 2019-10-31 11:12:26,982 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:12:37,001 DEBUG Dhanu => PING(20) 2019-10-31 11:12:46,991 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:12:57,021 DEBUG Dhanu => PING(20) 2019-10-31 11:13:07,001 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:13:17,044 DEBUG Dhanu => PING(20) 2019-10-31 11:13:27,008 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:13:37,063 DEBUG Dhanu => PING(20) 2019-10-31 11:13:47,015 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:13:57,080 DEBUG Dhanu => PING(20) 2019-10-31 11:14:07,023 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:14:17,099 DEBUG Dhanu => PING(20) 2019-10-31 11:14:27,030 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:14:37,116 DEBUG Dhanu => PING(20) 2019-10-31 11:14:47,038 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:14:57,131 DEBUG Dhanu => PING(20) 2019-10-31 11:15:07,045 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:15:17,153 DEBUG Dhanu => PING(20) 2019-10-31 11:15:27,053 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:15:37,177 DEBUG Dhanu => PING(20) 2019-10-31 11:15:47,061 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:15:57,197 DEBUG Dhanu => PING(20) 2019-10-31 11:16:07,069 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:16:17,218 DEBUG Dhanu => PING(20) 2019-10-31 11:16:27,077 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:16:37,237 DEBUG Dhanu => PING(20) 2019-10-31 11:16:47,086 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:16:57,256 DEBUG Dhanu => PING(20) 2019-10-31 11:17:07,093 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:17:17,272 DEBUG Dhanu => PING(20) 2019-10-31 11:17:27,100 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:17:37,292 DEBUG Dhanu => PING(20) 2019-10-31 11:17:47,108 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:17:57,314 DEBUG Dhanu => PING(20) 2019-10-31 11:18:07,115 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:18:17,332 DEBUG Dhanu => PING(20) 2019-10-31 11:18:27,124 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:18:37,348 DEBUG Dhanu => PING(20) 2019-10-31 11:18:47,132 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:18:57,364 DEBUG Dhanu => PING(20) 2019-10-31 11:19:07,140 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:19:17,381 DEBUG Dhanu => PING(20) 2019-10-31 11:19:27,146 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:19:37,400 DEBUG Dhanu => PING(20) 2019-10-31 11:19:47,154 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:19:57,419 DEBUG Dhanu => PING(20) 2019-10-31 11:20:07,158 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:20:17,439 DEBUG Dhanu => PING(20) 2019-10-31 11:20:27,166 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:20:37,458 DEBUG Dhanu => PING(20) 2019-10-31 11:20:47,174 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:20:57,480 DEBUG Dhanu => PING(20) 2019-10-31 11:21:07,182 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:21:17,502 DEBUG Dhanu => PING(20) 2019-10-31 11:21:27,189 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:21:37,524 DEBUG Dhanu => PING(20) 2019-10-31 11:21:47,196 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:21:57,544 DEBUG Dhanu => PING(20) 2019-10-31 11:22:07,205 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:22:17,563 DEBUG Dhanu => PING(20) 2019-10-31 11:22:27,215 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:22:37,587 DEBUG Dhanu => PING(20) 2019-10-31 11:22:47,223 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:22:57,604 DEBUG Dhanu => PING(20) 2019-10-31 11:23:07,232 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:23:17,625 DEBUG Dhanu => PING(20) 2019-10-31 11:23:27,241 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:23:37,650 DEBUG Dhanu => PING(20) 2019-10-31 11:23:47,250 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:23:57,668 DEBUG Dhanu => PING(20) 2019-10-31 11:24:07,260 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:24:17,684 DEBUG Dhanu => PING(20) 2019-10-31 11:24:27,269 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:24:37,699 DEBUG Dhanu => PING(20) 2019-10-31 11:24:47,277 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:24:57,717 DEBUG Dhanu => PING(20) 2019-10-31 11:25:07,283 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:25:17,739 DEBUG Dhanu => PING(20) 2019-10-31 11:25:27,292 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:25:37,761 DEBUG Dhanu => PING(20) 2019-10-31 11:25:47,300 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:25:57,780 DEBUG Dhanu => PING(20) 2019-10-31 11:26:07,306 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:26:17,795 DEBUG Dhanu => PING(20) 2019-10-31 11:26:27,316 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:26:37,811 DEBUG Dhanu => PING(20) 2019-10-31 11:26:47,325 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:26:57,833 DEBUG Dhanu => PING(20) 2019-10-31 11:27:07,332 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:27:17,851 DEBUG Dhanu => PING(20) 2019-10-31 11:27:27,339 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:27:37,867 DEBUG Dhanu => PING(20) 2019-10-31 11:27:47,344 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:27:57,891 DEBUG Dhanu => PING(20) 2019-10-31 11:28:07,352 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:28:17,911 DEBUG Dhanu => PING(20) 2019-10-31 11:28:27,360 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:28:37,935 DEBUG Dhanu => PING(20) 2019-10-31 11:28:47,368 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:28:57,952 DEBUG Dhanu => PING(20) 2019-10-31 11:29:07,376 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:29:17,971 DEBUG Dhanu => PING(20) 2019-10-31 11:29:27,385 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:29:37,988 DEBUG Dhanu => PING(20) 2019-10-31 11:29:47,393 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:29:58,004 DEBUG Dhanu => PING(20) 2019-10-31 11:30:07,401 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:30:18,019 DEBUG Dhanu => PING(20) 2019-10-31 11:30:27,411 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:30:38,040 DEBUG Dhanu => PING(20) 2019-10-31 11:30:47,418 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:30:58,055 DEBUG Dhanu => PING(20) 2019-10-31 11:31:07,425 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:31:18,077 DEBUG Dhanu => PING(20) 2019-10-31 11:31:27,432 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:31:38,101 DEBUG Dhanu => PING(20) 2019-10-31 11:31:47,441 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:31:58,126 DEBUG Dhanu => PING(20) 2019-10-31 11:32:07,447 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:32:18,148 DEBUG Dhanu => PING(20) 2019-10-31 11:32:27,454 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:32:38,171 DEBUG Dhanu => PING(20) 2019-10-31 11:32:47,462 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:32:58,195 DEBUG Dhanu => PING(20) 2019-10-31 11:33:07,468 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:33:18,217 DEBUG Dhanu => PING(20) 2019-10-31 11:33:27,476 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:33:38,241 DEBUG Dhanu => PING(20) 2019-10-31 11:33:47,484 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:33:58,265 DEBUG Dhanu => PING(20) 2019-10-31 11:34:07,492 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:34:18,288 DEBUG Dhanu => PING(20) 2019-10-31 11:34:27,500 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:34:38,313 DEBUG Dhanu => PING(20) 2019-10-31 11:34:47,508 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:34:58,337 DEBUG Dhanu => PING(20) 2019-10-31 11:35:07,517 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:35:18,358 DEBUG Dhanu => PING(20) 2019-10-31 11:35:27,525 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:35:38,379 DEBUG Dhanu => PING(20) 2019-10-31 11:35:47,533 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:35:58,396 DEBUG Dhanu => PING(20) 2019-10-31 11:36:07,541 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:36:18,416 DEBUG Dhanu => PING(20) 2019-10-31 11:36:27,548 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:36:38,440 DEBUG Dhanu => PING(20) 2019-10-31 11:36:47,555 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:36:58,456 DEBUG Dhanu => PING(20) 2019-10-31 11:37:07,197 INFO [POLL] Received a signal, leaving 2019-10-31 11:37:07,198 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 11:37:17,430 INFO [INIT] Dropping privileges 2019-10-31 11:37:17,430 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 11:37:17,430 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 11:37:17,432 INFO [INIT] Marking all workers as offline 2019-10-31 11:37:17,480 INFO [INIT] Starting encryption 2019-10-31 11:37:17,482 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-10-31 11:37:17,496 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-10-31 11:37:17,496 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-10-31 11:37:17,509 INFO [INIT] LAVA master has started. 2019-10-31 11:37:17,509 INFO [INIT] Using protocol version 3 2019-10-31 11:37:17,709 INFO Dhanu => HELLO 2019-10-31 11:37:17,709 WARNING New dispatcher 2019-10-31 11:37:22,715 DEBUG Dhanu => PING(20) 2019-10-31 11:37:37,523 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:37:42,731 DEBUG Dhanu => PING(20) 2019-10-31 11:37:57,536 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:38:02,752 DEBUG Dhanu => PING(20) 2019-10-31 11:38:17,550 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:38:22,773 DEBUG Dhanu => PING(20) 2019-10-31 11:38:37,558 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:38:42,795 DEBUG Dhanu => PING(20) 2019-10-31 11:38:57,565 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:39:02,815 DEBUG Dhanu => PING(20) 2019-10-31 11:39:17,574 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:39:22,831 DEBUG Dhanu => PING(20) 2019-10-31 11:39:37,586 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:39:42,847 DEBUG Dhanu => PING(20) 2019-10-31 11:39:57,601 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:40:02,868 DEBUG Dhanu => PING(20) 2019-10-31 11:40:17,614 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:40:22,892 DEBUG Dhanu => PING(20) 2019-10-31 11:40:37,624 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:40:42,911 DEBUG Dhanu => PING(20) 2019-10-31 11:40:57,638 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:41:02,928 DEBUG Dhanu => PING(20) 2019-10-31 11:41:17,649 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:41:22,945 DEBUG Dhanu => PING(20) 2019-10-31 11:41:37,663 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:41:42,963 DEBUG Dhanu => PING(20) 2019-10-31 11:41:57,674 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:42:02,979 DEBUG Dhanu => PING(20) 2019-10-31 11:42:17,687 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:42:22,996 DEBUG Dhanu => PING(20) 2019-10-31 11:42:37,702 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:42:43,013 DEBUG Dhanu => PING(20) 2019-10-31 11:42:57,714 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:43:03,031 DEBUG Dhanu => PING(20) 2019-10-31 11:43:17,726 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:43:23,057 DEBUG Dhanu => PING(20) 2019-10-31 11:43:37,741 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:43:43,081 DEBUG Dhanu => PING(20) 2019-10-31 11:43:57,756 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:44:03,100 DEBUG Dhanu => PING(20) 2019-10-31 11:44:17,770 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:44:23,120 DEBUG Dhanu => PING(20) 2019-10-31 11:44:37,825 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:44:43,136 DEBUG Dhanu => PING(20) 2019-10-31 11:44:57,832 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:45:03,156 DEBUG Dhanu => PING(20) 2019-10-31 11:45:17,839 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:45:23,177 DEBUG Dhanu => PING(20) 2019-10-31 11:45:37,847 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:45:43,199 DEBUG Dhanu => PING(20) 2019-10-31 11:45:57,855 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:46:03,220 DEBUG Dhanu => PING(20) 2019-10-31 11:46:17,863 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:46:23,239 DEBUG Dhanu => PING(20) 2019-10-31 11:46:37,870 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:46:43,259 DEBUG Dhanu => PING(20) 2019-10-31 11:46:57,877 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:47:03,279 DEBUG Dhanu => PING(20) 2019-10-31 11:47:17,884 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:47:23,299 DEBUG Dhanu => PING(20) 2019-10-31 11:47:37,891 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:47:43,315 DEBUG Dhanu => PING(20) 2019-10-31 11:47:57,899 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:48:03,335 DEBUG Dhanu => PING(20) 2019-10-31 11:48:17,908 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:48:23,355 DEBUG Dhanu => PING(20) 2019-10-31 11:48:37,919 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:48:43,371 DEBUG Dhanu => PING(20) 2019-10-31 11:48:57,933 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:49:03,395 DEBUG Dhanu => PING(20) 2019-10-31 11:49:17,947 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:49:23,416 DEBUG Dhanu => PING(20) 2019-10-31 11:49:37,958 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:49:43,435 DEBUG Dhanu => PING(20) 2019-10-31 11:49:57,972 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:50:03,459 DEBUG Dhanu => PING(20) 2019-10-31 11:50:17,987 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:50:23,485 DEBUG Dhanu => PING(20) 2019-10-31 11:50:38,001 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:50:43,508 DEBUG Dhanu => PING(20) 2019-10-31 11:50:58,016 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:51:03,533 DEBUG Dhanu => PING(20) 2019-10-31 11:51:18,030 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:51:23,556 DEBUG Dhanu => PING(20) 2019-10-31 11:51:38,044 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:51:43,581 DEBUG Dhanu => PING(20) 2019-10-31 11:51:58,058 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:52:03,604 DEBUG Dhanu => PING(20) 2019-10-31 11:52:18,070 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:52:23,627 DEBUG Dhanu => PING(20) 2019-10-31 11:52:38,085 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:52:43,647 DEBUG Dhanu => PING(20) 2019-10-31 11:52:58,092 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:53:03,668 DEBUG Dhanu => PING(20) 2019-10-31 11:53:18,105 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:53:23,693 DEBUG Dhanu => PING(20) 2019-10-31 11:53:38,119 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:53:43,716 DEBUG Dhanu => PING(20) 2019-10-31 11:53:58,131 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:54:03,740 DEBUG Dhanu => PING(20) 2019-10-31 11:54:18,145 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:54:23,763 DEBUG Dhanu => PING(20) 2019-10-31 11:54:38,155 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:54:43,787 DEBUG Dhanu => PING(20) 2019-10-31 11:54:58,169 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:55:03,812 DEBUG Dhanu => PING(20) 2019-10-31 11:55:18,183 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:55:23,836 DEBUG Dhanu => PING(20) 2019-10-31 11:55:38,197 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:55:43,861 DEBUG Dhanu => PING(20) 2019-10-31 11:55:58,211 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:56:03,880 DEBUG Dhanu => PING(20) 2019-10-31 11:56:18,225 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:56:23,903 DEBUG Dhanu => PING(20) 2019-10-31 11:56:38,239 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:56:43,925 DEBUG Dhanu => PING(20) 2019-10-31 11:56:58,250 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:57:03,948 DEBUG Dhanu => PING(20) 2019-10-31 11:57:18,264 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:57:23,972 DEBUG Dhanu => PING(20) 2019-10-31 11:57:38,279 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:57:43,993 DEBUG Dhanu => PING(20) 2019-10-31 11:57:58,291 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:58:04,015 DEBUG Dhanu => PING(20) 2019-10-31 11:58:18,299 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:58:24,036 DEBUG Dhanu => PING(20) 2019-10-31 11:58:38,307 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:58:44,053 DEBUG Dhanu => PING(20) 2019-10-31 11:58:58,321 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:59:04,072 DEBUG Dhanu => PING(20) 2019-10-31 11:59:18,331 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:59:24,092 DEBUG Dhanu => PING(20) 2019-10-31 11:59:38,343 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 11:59:44,213 DEBUG Dhanu => PING(20) 2019-10-31 11:59:58,356 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:00:04,236 DEBUG Dhanu => PING(20) 2019-10-31 12:00:18,370 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:00:24,260 DEBUG Dhanu => PING(20) 2019-10-31 12:00:38,384 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:00:44,283 DEBUG Dhanu => PING(20) 2019-10-31 12:00:58,398 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:01:04,307 DEBUG Dhanu => PING(20) 2019-10-31 12:01:18,411 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:01:24,332 DEBUG Dhanu => PING(20) 2019-10-31 12:01:38,424 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:01:44,353 DEBUG Dhanu => PING(20) 2019-10-31 12:01:58,439 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:02:04,378 DEBUG Dhanu => PING(20) 2019-10-31 12:02:18,453 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:02:24,400 DEBUG Dhanu => PING(20) 2019-10-31 12:02:38,461 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:02:44,425 DEBUG Dhanu => PING(20) 2019-10-31 12:02:58,470 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:03:04,449 DEBUG Dhanu => PING(20) 2019-10-31 12:03:18,478 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:03:24,474 DEBUG Dhanu => PING(20) 2019-10-31 12:03:38,486 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:03:44,498 DEBUG Dhanu => PING(20) 2019-10-31 12:03:58,498 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:04:04,523 DEBUG Dhanu => PING(20) 2019-10-31 12:04:18,512 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:04:24,547 DEBUG Dhanu => PING(20) 2019-10-31 12:04:38,522 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:04:44,564 DEBUG Dhanu => PING(20) 2019-10-31 12:04:58,536 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:05:04,587 DEBUG Dhanu => PING(20) 2019-10-31 12:05:18,549 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:05:24,605 DEBUG Dhanu => PING(20) 2019-10-31 12:05:38,556 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:05:44,629 DEBUG Dhanu => PING(20) 2019-10-31 12:05:58,564 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:06:04,654 DEBUG Dhanu => PING(20) 2019-10-31 12:06:18,578 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:06:24,678 DEBUG Dhanu => PING(20) 2019-10-31 12:06:38,591 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:06:44,703 DEBUG Dhanu => PING(20) 2019-10-31 12:06:58,603 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:07:04,723 DEBUG Dhanu => PING(20) 2019-10-31 12:07:18,615 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:07:24,746 DEBUG Dhanu => PING(20) 2019-10-31 12:07:38,629 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:07:44,760 DEBUG Dhanu => PING(20) 2019-10-31 12:07:58,642 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:08:04,782 DEBUG Dhanu => PING(20) 2019-10-31 12:08:18,656 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:08:24,802 DEBUG Dhanu => PING(20) 2019-10-31 12:08:38,670 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:08:44,848 DEBUG Dhanu => PING(20) 2019-10-31 12:08:58,684 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:09:04,872 DEBUG Dhanu => PING(20) 2019-10-31 12:09:18,692 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:09:24,888 DEBUG Dhanu => PING(20) 2019-10-31 12:09:38,706 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:09:44,912 DEBUG Dhanu => PING(20) 2019-10-31 12:09:58,719 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:10:04,932 DEBUG Dhanu => PING(20) 2019-10-31 12:10:18,727 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:10:24,957 DEBUG Dhanu => PING(20) 2019-10-31 12:10:38,734 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:10:44,980 DEBUG Dhanu => PING(20) 2019-10-31 12:10:58,744 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:11:05,005 DEBUG Dhanu => PING(20) 2019-10-31 12:11:18,758 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:11:25,027 DEBUG Dhanu => PING(20) 2019-10-31 12:11:38,771 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:11:45,052 DEBUG Dhanu => PING(20) 2019-10-31 12:11:58,784 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:12:05,073 DEBUG Dhanu => PING(20) 2019-10-31 12:12:18,797 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:12:25,115 DEBUG Dhanu => PING(20) 2019-10-31 12:12:38,806 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:12:45,135 DEBUG Dhanu => PING(20) 2019-10-31 12:12:58,815 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:13:05,160 DEBUG Dhanu => PING(20) 2019-10-31 12:13:18,829 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:13:25,185 DEBUG Dhanu => PING(20) 2019-10-31 12:13:38,843 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:13:45,205 DEBUG Dhanu => PING(20) 2019-10-31 12:13:58,856 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:14:05,229 DEBUG Dhanu => PING(20) 2019-10-31 12:14:18,866 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:14:25,252 DEBUG Dhanu => PING(20) 2019-10-31 12:14:38,873 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:14:45,276 DEBUG Dhanu => PING(20) 2019-10-31 12:14:54,782 INFO [POLL] Received a signal, leaving 2019-10-31 12:14:54,782 INFO [CLOSE] Closing the controler socket and dropping messages 2019-10-31 12:14:59,272 INFO [INIT] Dropping privileges 2019-10-31 12:14:59,272 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-10-31 12:14:59,272 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-10-31 12:14:59,274 INFO [INIT] Marking all workers as offline 2019-10-31 12:14:59,344 INFO [INIT] Starting encryption 2019-10-31 12:14:59,346 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-10-31 12:14:59,346 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-10-31 12:14:59,347 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-10-31 12:14:59,362 INFO [INIT] LAVA master has started. 2019-10-31 12:14:59,363 INFO [INIT] Using protocol version 3 2019-10-31 12:15:01,895 INFO Dhanu => HELLO 2019-10-31 12:15:01,896 WARNING New dispatcher 2019-10-31 12:15:06,901 DEBUG Dhanu => PING(20) 2019-10-31 12:15:19,374 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:15:26,919 DEBUG Dhanu => PING(20) 2019-10-31 12:15:39,382 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:15:46,935 DEBUG Dhanu => PING(20) 2019-10-31 12:15:59,415 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:16:07,182 DEBUG Dhanu => PING(20) 2019-10-31 12:16:19,522 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:16:27,200 DEBUG Dhanu => PING(20) 2019-10-31 12:16:39,529 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:16:47,219 DEBUG Dhanu => PING(20) 2019-10-31 12:16:59,537 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:17:07,236 DEBUG Dhanu => PING(20) 2019-10-31 12:17:19,545 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:17:27,251 DEBUG Dhanu => PING(20) 2019-10-31 12:17:39,551 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:17:47,267 DEBUG Dhanu => PING(20) 2019-10-31 12:17:59,588 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:18:07,291 DEBUG Dhanu => PING(20) 2019-10-31 12:18:19,596 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:18:27,309 DEBUG Dhanu => PING(20) 2019-10-31 12:18:39,606 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:18:47,329 DEBUG Dhanu => PING(20) 2019-10-31 12:18:59,614 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:19:07,351 DEBUG Dhanu => PING(20) 2019-10-31 12:19:19,622 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:19:27,371 DEBUG Dhanu => PING(20) 2019-10-31 12:19:39,630 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:19:47,395 DEBUG Dhanu => PING(20) 2019-10-31 12:19:59,639 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:20:07,416 DEBUG Dhanu => PING(20) 2019-10-31 12:20:19,647 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:20:27,434 DEBUG Dhanu => PING(20) 2019-10-31 12:20:39,655 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:20:47,456 DEBUG Dhanu => PING(20) 2019-10-31 12:20:59,662 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:21:07,480 DEBUG Dhanu => PING(20) 2019-10-31 12:21:19,670 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:21:27,501 DEBUG Dhanu => PING(20) 2019-10-31 12:21:39,678 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:21:47,525 DEBUG Dhanu => PING(20) 2019-10-31 12:21:59,686 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:22:07,547 DEBUG Dhanu => PING(20) 2019-10-31 12:22:19,695 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:22:27,567 DEBUG Dhanu => PING(20) 2019-10-31 12:22:39,700 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:22:47,584 DEBUG Dhanu => PING(20) 2019-10-31 12:22:59,708 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:23:07,603 DEBUG Dhanu => PING(20) 2019-10-31 12:23:19,716 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:23:27,624 DEBUG Dhanu => PING(20) 2019-10-31 12:23:39,723 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:23:47,648 DEBUG Dhanu => PING(20) 2019-10-31 12:23:59,731 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:24:07,668 DEBUG Dhanu => PING(20) 2019-10-31 12:24:19,738 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:24:27,687 DEBUG Dhanu => PING(20) 2019-10-31 12:24:39,746 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:24:47,711 DEBUG Dhanu => PING(20) 2019-10-31 12:24:59,753 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:25:07,729 DEBUG Dhanu => PING(20) 2019-10-31 12:25:19,762 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:25:27,751 DEBUG Dhanu => PING(20) 2019-10-31 12:25:39,770 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:25:47,775 DEBUG Dhanu => PING(20) 2019-10-31 12:25:59,777 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:26:07,800 DEBUG Dhanu => PING(20) 2019-10-31 12:26:19,785 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:26:27,820 DEBUG Dhanu => PING(20) 2019-10-31 12:26:39,794 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:26:47,844 DEBUG Dhanu => PING(20) 2019-10-31 12:26:59,801 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:27:07,868 DEBUG Dhanu => PING(20) 2019-10-31 12:27:19,809 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:27:27,893 DEBUG Dhanu => PING(20) 2019-10-31 12:27:39,817 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:27:47,912 DEBUG Dhanu => PING(20) 2019-10-31 12:27:59,824 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:28:07,936 DEBUG Dhanu => PING(20) 2019-10-31 12:28:19,832 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:28:27,960 DEBUG Dhanu => PING(20) 2019-10-31 12:28:39,840 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:28:47,984 DEBUG Dhanu => PING(20) 2019-10-31 12:28:59,847 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:29:08,007 DEBUG Dhanu => PING(20) 2019-10-31 12:29:19,854 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:29:28,024 DEBUG Dhanu => PING(20) 2019-10-31 12:29:39,862 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:29:48,048 DEBUG Dhanu => PING(20) 2019-10-31 12:29:59,871 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:30:08,070 DEBUG Dhanu => PING(20) 2019-10-31 12:30:19,878 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:30:28,094 DEBUG Dhanu => PING(20) 2019-10-31 12:30:39,887 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:30:48,113 DEBUG Dhanu => PING(20) 2019-10-31 12:30:59,893 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:31:08,134 DEBUG Dhanu => PING(20) 2019-10-31 12:31:19,901 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:31:28,155 DEBUG Dhanu => PING(20) 2019-10-31 12:31:39,909 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:31:48,176 DEBUG Dhanu => PING(20) 2019-10-31 12:31:59,917 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:32:08,196 DEBUG Dhanu => PING(20) 2019-10-31 12:32:19,925 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:32:28,221 DEBUG Dhanu => PING(20) 2019-10-31 12:32:39,934 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:32:48,244 DEBUG Dhanu => PING(20) 2019-10-31 12:32:59,941 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:33:08,269 DEBUG Dhanu => PING(20) 2019-10-31 12:33:19,949 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:33:28,294 DEBUG Dhanu => PING(20) 2019-10-31 12:33:39,957 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:33:48,316 DEBUG Dhanu => PING(20) 2019-10-31 12:33:59,966 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:34:08,340 DEBUG Dhanu => PING(20) 2019-10-31 12:34:19,973 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:34:28,365 DEBUG Dhanu => PING(20) 2019-10-31 12:34:39,981 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:34:48,387 DEBUG Dhanu => PING(20) 2019-10-31 12:34:59,989 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:35:08,408 DEBUG Dhanu => PING(20) 2019-10-31 12:35:19,998 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:35:28,433 DEBUG Dhanu => PING(20) 2019-10-31 12:35:40,006 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:35:48,457 DEBUG Dhanu => PING(20) 2019-10-31 12:36:00,014 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:36:08,482 DEBUG Dhanu => PING(20) 2019-10-31 12:36:20,017 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:36:28,507 DEBUG Dhanu => PING(20) 2019-10-31 12:36:40,025 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:36:48,528 DEBUG Dhanu => PING(20) 2019-10-31 12:37:00,033 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:37:08,552 DEBUG Dhanu => PING(20) 2019-10-31 12:37:20,038 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:37:28,575 DEBUG Dhanu => PING(20) 2019-10-31 12:37:40,046 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:37:48,599 DEBUG Dhanu => PING(20) 2019-10-31 12:38:00,053 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:38:08,620 DEBUG Dhanu => PING(20) 2019-10-31 12:38:20,060 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:38:28,641 DEBUG Dhanu => PING(20) 2019-10-31 12:38:40,067 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:38:48,664 DEBUG Dhanu => PING(20) 2019-10-31 12:39:00,074 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:39:08,684 DEBUG Dhanu => PING(20) 2019-10-31 12:39:20,082 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:39:28,708 DEBUG Dhanu => PING(20) 2019-10-31 12:39:40,090 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:39:48,729 DEBUG Dhanu => PING(20) 2019-10-31 12:40:00,101 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:40:08,753 DEBUG Dhanu => PING(20) 2019-10-31 12:40:20,112 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:40:28,778 DEBUG Dhanu => PING(20) 2019-10-31 12:40:40,120 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:40:48,803 DEBUG Dhanu => PING(20) 2019-10-31 12:41:00,127 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:41:08,827 DEBUG Dhanu => PING(20) 2019-10-31 12:41:20,135 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:41:28,850 DEBUG Dhanu => PING(20) 2019-10-31 12:41:40,146 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:41:48,870 DEBUG Dhanu => PING(20) 2019-10-31 12:42:00,154 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:42:08,891 DEBUG Dhanu => PING(20) 2019-10-31 12:42:20,162 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:42:28,911 DEBUG Dhanu => PING(20) 2019-10-31 12:42:40,171 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:42:48,933 DEBUG Dhanu => PING(20) 2019-10-31 12:43:00,179 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:43:08,954 DEBUG Dhanu => PING(20) 2019-10-31 12:43:20,186 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:43:28,976 DEBUG Dhanu => PING(20) 2019-10-31 12:43:40,198 WARNING lava-logs is offline: can't schedule jobs 2019-10-31 12:43:49,001 DEBUG Dhanu => PING(20) 2019-11-04 04:52:14,104 ERROR [STATE] Dispatcher goes OFFLINE 2019-11-04 04:52:14,887 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:52:14,965 DEBUG Dhanu => PING(20) 2019-11-04 04:52:34,965 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:52:34,983 DEBUG Dhanu => PING(20) 2019-11-04 04:52:54,968 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:52:54,999 DEBUG Dhanu => PING(20) 2019-11-04 04:53:14,975 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:53:15,015 DEBUG Dhanu => PING(20) 2019-11-04 04:53:34,982 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:53:35,039 DEBUG Dhanu => PING(20) 2019-11-04 04:53:54,989 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:53:55,060 DEBUG Dhanu => PING(20) 2019-11-04 04:54:14,996 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:54:15,080 DEBUG Dhanu => PING(20) 2019-11-04 04:54:35,003 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:54:35,102 DEBUG Dhanu => PING(20) 2019-11-04 04:54:55,010 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:54:55,125 DEBUG Dhanu => PING(20) 2019-11-04 04:55:15,017 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:55:15,145 DEBUG Dhanu => PING(20) 2019-11-04 04:55:35,024 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:55:35,165 DEBUG Dhanu => PING(20) 2019-11-04 04:55:55,032 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:55:55,189 DEBUG Dhanu => PING(20) 2019-11-04 04:56:15,039 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:56:15,212 DEBUG Dhanu => PING(20) 2019-11-04 04:56:35,047 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:56:35,231 DEBUG Dhanu => PING(20) 2019-11-04 04:56:55,055 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:56:55,252 DEBUG Dhanu => PING(20) 2019-11-04 04:57:15,062 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:57:15,276 DEBUG Dhanu => PING(20) 2019-11-04 04:57:35,066 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:57:35,299 DEBUG Dhanu => PING(20) 2019-11-04 04:57:55,073 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:57:55,323 DEBUG Dhanu => PING(20) 2019-11-04 04:58:15,080 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:58:15,347 DEBUG Dhanu => PING(20) 2019-11-04 04:58:35,088 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:58:35,365 DEBUG Dhanu => PING(20) 2019-11-04 04:58:55,096 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:58:55,389 DEBUG Dhanu => PING(20) 2019-11-04 04:59:15,103 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:59:15,414 DEBUG Dhanu => PING(20) 2019-11-04 04:59:35,110 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:59:35,436 DEBUG Dhanu => PING(20) 2019-11-04 04:59:55,117 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 04:59:55,461 DEBUG Dhanu => PING(20) 2019-11-04 05:00:15,124 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:00:15,481 DEBUG Dhanu => PING(20) 2019-11-04 05:00:35,132 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:00:35,504 DEBUG Dhanu => PING(20) 2019-11-04 05:00:55,140 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:00:55,521 DEBUG Dhanu => PING(20) 2019-11-04 05:01:15,148 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:01:15,545 DEBUG Dhanu => PING(20) 2019-11-04 05:01:35,155 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:01:35,564 DEBUG Dhanu => PING(20) 2019-11-04 05:01:55,161 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:01:55,584 DEBUG Dhanu => PING(20) 2019-11-04 05:02:15,166 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:02:15,603 DEBUG Dhanu => PING(20) 2019-11-04 05:02:35,174 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:02:35,627 DEBUG Dhanu => PING(20) 2019-11-04 05:02:55,179 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:02:55,649 DEBUG Dhanu => PING(20) 2019-11-04 05:03:15,187 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:03:15,669 DEBUG Dhanu => PING(20) 2019-11-04 05:03:35,195 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:03:35,690 DEBUG Dhanu => PING(20) 2019-11-04 05:03:55,202 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:03:55,714 DEBUG Dhanu => PING(20) 2019-11-04 05:04:15,209 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:04:15,736 DEBUG Dhanu => PING(20) 2019-11-04 05:04:35,216 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:04:35,755 DEBUG Dhanu => PING(20) 2019-11-04 05:04:55,223 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:04:55,777 DEBUG Dhanu => PING(20) 2019-11-04 05:05:15,230 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:05:15,798 DEBUG Dhanu => PING(20) 2019-11-04 05:05:35,236 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:05:35,816 DEBUG Dhanu => PING(20) 2019-11-04 05:05:55,243 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:05:55,836 DEBUG Dhanu => PING(20) 2019-11-04 05:06:15,249 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:06:15,855 DEBUG Dhanu => PING(20) 2019-11-04 05:06:35,256 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:06:35,876 DEBUG Dhanu => PING(20) 2019-11-04 05:06:55,263 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:06:55,899 DEBUG Dhanu => PING(20) 2019-11-04 05:07:15,273 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:07:15,923 DEBUG Dhanu => PING(20) 2019-11-04 05:07:35,281 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:07:35,945 DEBUG Dhanu => PING(20) 2019-11-04 05:07:55,288 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:07:55,965 DEBUG Dhanu => PING(20) 2019-11-04 05:08:15,297 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:08:15,983 DEBUG Dhanu => PING(20) 2019-11-04 05:08:35,304 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:08:36,008 DEBUG Dhanu => PING(20) 2019-11-04 05:08:55,307 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:08:56,028 DEBUG Dhanu => PING(20) 2019-11-04 05:09:15,315 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:09:16,050 DEBUG Dhanu => PING(20) 2019-11-04 05:09:35,323 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:09:36,072 DEBUG Dhanu => PING(20) 2019-11-04 05:09:55,331 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:09:56,096 DEBUG Dhanu => PING(20) 2019-11-04 05:10:15,339 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:10:16,117 DEBUG Dhanu => PING(20) 2019-11-04 05:10:35,346 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:10:36,141 DEBUG Dhanu => PING(20) 2019-11-04 05:10:55,353 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:10:56,166 DEBUG Dhanu => PING(20) 2019-11-04 05:11:15,361 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:11:16,187 DEBUG Dhanu => PING(20) 2019-11-04 05:11:35,369 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:11:36,211 DEBUG Dhanu => PING(20) 2019-11-04 05:11:55,377 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:11:56,235 DEBUG Dhanu => PING(20) 2019-11-04 05:12:15,383 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:12:16,251 DEBUG Dhanu => PING(20) 2019-11-04 05:12:35,389 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:12:36,267 DEBUG Dhanu => PING(20) 2019-11-04 05:12:55,413 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:12:56,288 DEBUG Dhanu => PING(20) 2019-11-04 05:13:15,421 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:13:16,307 DEBUG Dhanu => PING(20) 2019-11-04 05:13:35,428 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:13:36,332 DEBUG Dhanu => PING(20) 2019-11-04 05:13:55,435 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:13:56,356 DEBUG Dhanu => PING(20) 2019-11-04 05:14:15,442 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:14:16,381 DEBUG Dhanu => PING(20) 2019-11-04 05:14:35,450 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:14:36,404 DEBUG Dhanu => PING(20) 2019-11-04 05:14:55,459 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:14:56,424 DEBUG Dhanu => PING(20) 2019-11-04 05:15:15,466 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:15:16,448 DEBUG Dhanu => PING(20) 2019-11-04 05:15:35,474 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:15:36,472 DEBUG Dhanu => PING(20) 2019-11-04 05:15:55,481 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:15:56,496 DEBUG Dhanu => PING(20) 2019-11-04 05:16:15,489 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:16:16,521 DEBUG Dhanu => PING(20) 2019-11-04 05:16:35,496 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:16:36,544 DEBUG Dhanu => PING(20) 2019-11-04 05:16:55,504 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:16:56,563 DEBUG Dhanu => PING(20) 2019-11-04 05:17:15,511 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:17:16,585 DEBUG Dhanu => PING(20) 2019-11-04 05:17:35,519 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:17:36,607 DEBUG Dhanu => PING(20) 2019-11-04 05:17:55,527 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:17:56,629 DEBUG Dhanu => PING(20) 2019-11-04 05:18:15,534 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:18:16,651 DEBUG Dhanu => PING(20) 2019-11-04 05:18:35,542 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:18:36,676 DEBUG Dhanu => PING(20) 2019-11-04 05:18:55,550 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:18:56,695 DEBUG Dhanu => PING(20) 2019-11-04 05:19:15,558 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:19:16,720 DEBUG Dhanu => PING(20) 2019-11-04 05:19:35,566 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:19:36,744 DEBUG Dhanu => PING(20) 2019-11-04 05:19:55,574 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:19:56,768 DEBUG Dhanu => PING(20) 2019-11-04 05:20:15,583 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:20:16,792 DEBUG Dhanu => PING(20) 2019-11-04 05:20:35,590 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:20:36,816 DEBUG Dhanu => PING(20) 2019-11-04 05:20:55,598 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:20:56,840 DEBUG Dhanu => PING(20) 2019-11-04 05:21:15,607 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:21:16,865 DEBUG Dhanu => PING(20) 2019-11-04 05:21:35,614 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:21:36,889 DEBUG Dhanu => PING(20) 2019-11-04 05:21:55,622 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:21:56,913 DEBUG Dhanu => PING(20) 2019-11-04 05:22:15,630 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:22:16,935 DEBUG Dhanu => PING(20) 2019-11-04 05:22:35,638 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:22:36,957 DEBUG Dhanu => PING(20) 2019-11-04 05:22:55,645 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:22:56,980 DEBUG Dhanu => PING(20) 2019-11-04 05:23:15,653 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:23:16,999 DEBUG Dhanu => PING(20) 2019-11-04 05:23:35,656 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:23:37,016 DEBUG Dhanu => PING(20) 2019-11-04 05:23:55,663 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:23:57,031 DEBUG Dhanu => PING(20) 2019-11-04 05:24:15,671 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:24:17,051 DEBUG Dhanu => PING(20) 2019-11-04 05:24:17,076 INFO scheduling health checks: 2019-11-04 05:24:18,132 INFO scheduling jobs: 2019-11-04 05:24:18,137 DEBUG - qemu 2019-11-04 05:24:19,018 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 05:24:19,019 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 799, in main_loop jobs = schedule(self.logger, self.events["available_dt"]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 05:24:19,275 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 05:24:24,774 INFO [INIT] Dropping privileges 2019-11-04 05:24:24,774 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 05:24:24,774 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 05:24:24,776 INFO [INIT] Marking all workers as offline 2019-11-04 05:24:24,811 INFO [INIT] Starting encryption 2019-11-04 05:24:24,813 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 05:24:24,847 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 05:24:24,848 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 05:24:24,867 INFO [INIT] LAVA master has started. 2019-11-04 05:24:24,868 INFO [INIT] Using protocol version 3 2019-11-04 05:24:37,068 DEBUG Dhanu => PING(20) 2019-11-04 05:24:37,068 WARNING Unknown dispatcher (server crashed) 2019-11-04 05:24:44,875 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:24:57,085 DEBUG Dhanu => PING(20) 2019-11-04 05:25:04,884 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:25:17,104 DEBUG Dhanu => PING(20) 2019-11-04 05:25:24,890 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:25:37,119 DEBUG Dhanu => PING(20) 2019-11-04 05:25:44,897 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:25:57,141 DEBUG Dhanu => PING(20) 2019-11-04 05:26:04,905 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:26:17,160 DEBUG Dhanu => PING(20) 2019-11-04 05:26:24,913 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:26:37,184 DEBUG Dhanu => PING(20) 2019-11-04 05:26:44,921 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:26:57,205 DEBUG Dhanu => PING(20) 2019-11-04 05:27:04,929 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:27:17,225 DEBUG Dhanu => PING(20) 2019-11-04 05:27:24,937 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:27:37,243 DEBUG Dhanu => PING(20) 2019-11-04 05:27:44,945 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:27:57,280 DEBUG Dhanu => PING(20) 2019-11-04 05:28:04,953 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:28:17,295 DEBUG Dhanu => PING(20) 2019-11-04 05:28:24,962 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:28:37,311 DEBUG Dhanu => PING(20) 2019-11-04 05:28:44,969 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:28:57,335 DEBUG Dhanu => PING(20) 2019-11-04 05:29:04,976 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:29:17,359 DEBUG Dhanu => PING(20) 2019-11-04 05:29:24,985 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:29:37,381 DEBUG Dhanu => PING(20) 2019-11-04 05:29:44,993 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:29:57,401 DEBUG Dhanu => PING(20) 2019-11-04 05:30:05,000 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:30:17,426 DEBUG Dhanu => PING(20) 2019-11-04 05:30:25,008 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:30:37,446 DEBUG Dhanu => PING(20) 2019-11-04 05:30:45,016 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:30:57,468 DEBUG Dhanu => PING(20) 2019-11-04 05:31:05,023 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:31:17,492 DEBUG Dhanu => PING(20) 2019-11-04 05:31:25,031 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:31:37,516 DEBUG Dhanu => PING(20) 2019-11-04 05:31:45,039 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:31:57,537 DEBUG Dhanu => PING(20) 2019-11-04 05:32:05,045 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:32:17,562 DEBUG Dhanu => PING(20) 2019-11-04 05:32:25,053 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:32:37,587 DEBUG Dhanu => PING(20) 2019-11-04 05:32:45,061 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:32:57,607 DEBUG Dhanu => PING(20) 2019-11-04 05:33:05,068 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:33:17,631 DEBUG Dhanu => PING(20) 2019-11-04 05:33:25,077 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:33:37,649 DEBUG Dhanu => PING(20) 2019-11-04 05:33:45,085 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:33:57,667 DEBUG Dhanu => PING(20) 2019-11-04 05:34:05,093 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:34:17,691 DEBUG Dhanu => PING(20) 2019-11-04 05:34:25,100 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:34:37,716 DEBUG Dhanu => PING(20) 2019-11-04 05:34:45,108 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:34:57,740 DEBUG Dhanu => PING(20) 2019-11-04 05:35:05,116 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:35:17,760 DEBUG Dhanu => PING(20) 2019-11-04 05:35:25,122 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:35:37,781 DEBUG Dhanu => PING(20) 2019-11-04 05:35:45,130 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:35:57,796 DEBUG Dhanu => PING(20) 2019-11-04 05:36:05,138 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:36:17,819 DEBUG Dhanu => PING(20) 2019-11-04 05:36:25,146 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:36:37,836 DEBUG Dhanu => PING(20) 2019-11-04 05:36:45,155 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:36:57,856 DEBUG Dhanu => PING(20) 2019-11-04 05:37:05,162 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:37:17,876 DEBUG Dhanu => PING(20) 2019-11-04 05:37:25,170 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:37:37,897 DEBUG Dhanu => PING(20) 2019-11-04 05:37:45,177 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:37:57,921 DEBUG Dhanu => PING(20) 2019-11-04 05:38:05,185 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:38:17,945 DEBUG Dhanu => PING(20) 2019-11-04 05:38:25,193 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:38:37,970 DEBUG Dhanu => PING(20) 2019-11-04 05:38:45,201 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:38:57,988 DEBUG Dhanu => PING(20) 2019-11-04 05:39:05,209 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:39:18,008 DEBUG Dhanu => PING(20) 2019-11-04 05:39:25,217 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:39:38,032 DEBUG Dhanu => PING(20) 2019-11-04 05:39:45,224 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:39:58,056 DEBUG Dhanu => PING(20) 2019-11-04 05:40:05,231 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:40:18,081 DEBUG Dhanu => PING(20) 2019-11-04 05:40:25,239 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:40:38,101 DEBUG Dhanu => PING(20) 2019-11-04 05:40:45,247 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:40:58,121 DEBUG Dhanu => PING(20) 2019-11-04 05:41:05,254 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:41:18,143 DEBUG Dhanu => PING(20) 2019-11-04 05:41:25,263 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:41:38,164 DEBUG Dhanu => PING(20) 2019-11-04 05:41:45,271 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:41:58,184 DEBUG Dhanu => PING(20) 2019-11-04 05:42:05,278 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:42:18,207 DEBUG Dhanu => PING(20) 2019-11-04 05:42:25,285 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:42:38,227 DEBUG Dhanu => PING(20) 2019-11-04 05:42:45,293 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:42:58,248 DEBUG Dhanu => PING(20) 2019-11-04 05:43:05,301 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:43:18,269 DEBUG Dhanu => PING(20) 2019-11-04 05:43:25,309 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:43:38,289 DEBUG Dhanu => PING(20) 2019-11-04 05:43:45,317 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:43:58,314 DEBUG Dhanu => PING(20) 2019-11-04 05:44:05,325 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:44:18,337 DEBUG Dhanu => PING(20) 2019-11-04 05:44:25,332 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:44:38,362 DEBUG Dhanu => PING(20) 2019-11-04 05:44:45,339 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:44:58,386 DEBUG Dhanu => PING(20) 2019-11-04 05:45:05,347 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:45:18,407 DEBUG Dhanu => PING(20) 2019-11-04 05:45:25,354 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:45:38,432 DEBUG Dhanu => PING(20) 2019-11-04 05:45:45,362 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:45:58,456 DEBUG Dhanu => PING(20) 2019-11-04 05:46:05,370 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:46:18,480 DEBUG Dhanu => PING(20) 2019-11-04 05:46:25,378 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:46:38,505 DEBUG Dhanu => PING(20) 2019-11-04 05:46:45,385 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:46:58,524 DEBUG Dhanu => PING(20) 2019-11-04 05:47:05,393 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:47:18,546 DEBUG Dhanu => PING(20) 2019-11-04 05:47:25,401 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:47:38,569 DEBUG Dhanu => PING(20) 2019-11-04 05:47:45,409 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:47:58,591 DEBUG Dhanu => PING(20) 2019-11-04 05:48:05,417 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:48:18,612 DEBUG Dhanu => PING(20) 2019-11-04 05:48:25,425 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:48:38,637 DEBUG Dhanu => PING(20) 2019-11-04 05:48:45,433 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:48:58,662 DEBUG Dhanu => PING(20) 2019-11-04 05:49:05,441 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:49:18,687 DEBUG Dhanu => PING(20) 2019-11-04 05:49:25,450 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:49:38,708 DEBUG Dhanu => PING(20) 2019-11-04 05:49:45,458 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:49:58,732 DEBUG Dhanu => PING(20) 2019-11-04 05:50:05,465 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:50:18,757 DEBUG Dhanu => PING(20) 2019-11-04 05:50:25,474 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:50:38,781 DEBUG Dhanu => PING(20) 2019-11-04 05:50:45,481 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:50:58,801 DEBUG Dhanu => PING(20) 2019-11-04 05:51:05,489 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:51:18,821 DEBUG Dhanu => PING(20) 2019-11-04 05:51:25,496 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:51:38,844 DEBUG Dhanu => PING(20) 2019-11-04 05:51:45,505 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:51:58,865 DEBUG Dhanu => PING(20) 2019-11-04 05:52:05,512 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:52:18,885 DEBUG Dhanu => PING(20) 2019-11-04 05:52:25,521 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:52:38,908 DEBUG Dhanu => PING(20) 2019-11-04 05:52:45,529 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:52:58,925 DEBUG Dhanu => PING(20) 2019-11-04 05:53:05,537 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:53:18,944 DEBUG Dhanu => PING(20) 2019-11-04 05:53:25,544 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:53:38,965 DEBUG Dhanu => PING(20) 2019-11-04 05:53:45,552 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:53:58,988 DEBUG Dhanu => PING(20) 2019-11-04 05:54:05,559 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:54:19,008 DEBUG Dhanu => PING(20) 2019-11-04 05:54:25,564 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:54:39,033 DEBUG Dhanu => PING(20) 2019-11-04 05:54:45,571 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:54:59,055 DEBUG Dhanu => PING(20) 2019-11-04 05:55:05,579 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:55:19,076 DEBUG Dhanu => PING(20) 2019-11-04 05:55:25,587 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:55:39,096 DEBUG Dhanu => PING(20) 2019-11-04 05:55:45,594 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:55:59,112 DEBUG Dhanu => PING(20) 2019-11-04 05:56:05,601 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:57:50,862 DEBUG Dhanu => PING(20) 2019-11-04 05:57:50,913 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:58:10,887 DEBUG Dhanu => PING(20) 2019-11-04 05:58:10,926 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:58:30,907 DEBUG Dhanu => PING(20) 2019-11-04 05:58:30,934 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:58:50,923 DEBUG Dhanu => PING(20) 2019-11-04 05:58:50,953 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:59:10,939 DEBUG Dhanu => PING(20) 2019-11-04 05:59:10,967 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:59:30,959 DEBUG Dhanu => PING(20) 2019-11-04 05:59:30,978 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 05:59:50,983 DEBUG Dhanu => PING(20) 2019-11-04 05:59:51,003 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:00:11,002 DEBUG Dhanu => PING(20) 2019-11-04 06:00:11,027 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:00:31,023 DEBUG Dhanu => PING(20) 2019-11-04 06:00:31,051 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:00:51,047 DEBUG Dhanu => PING(20) 2019-11-04 06:00:51,064 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:01:11,072 DEBUG Dhanu => PING(20) 2019-11-04 06:01:11,099 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:01:31,096 DEBUG Dhanu => PING(20) 2019-11-04 06:01:31,112 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:01:51,120 DEBUG Dhanu => PING(20) 2019-11-04 06:01:51,148 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:02:11,145 DEBUG Dhanu => PING(20) 2019-11-04 06:02:11,161 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:02:31,168 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:02:31,175 DEBUG Dhanu => PING(20) 2019-11-04 06:02:51,175 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:02:51,195 DEBUG Dhanu => PING(20) 2019-11-04 06:03:11,191 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:03:11,217 DEBUG Dhanu => PING(20) 2019-11-04 06:03:31,211 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:03:31,241 DEBUG Dhanu => PING(20) 2019-11-04 06:03:51,231 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:03:51,263 DEBUG Dhanu => PING(20) 2019-11-04 06:04:11,250 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:04:11,287 DEBUG Dhanu => PING(20) 2019-11-04 06:04:31,257 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:04:31,306 DEBUG Dhanu => PING(20) 2019-11-04 06:04:51,270 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:04:51,324 DEBUG Dhanu => PING(20) 2019-11-04 06:05:11,286 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:05:11,343 DEBUG Dhanu => PING(20) 2019-11-04 06:05:31,300 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:05:31,359 DEBUG Dhanu => PING(20) 2019-11-04 06:05:51,318 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:05:51,376 DEBUG Dhanu => PING(20) 2019-11-04 06:06:11,338 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:06:11,399 DEBUG Dhanu => PING(20) 2019-11-04 06:06:31,358 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:06:31,423 DEBUG Dhanu => PING(20) 2019-11-04 06:06:51,378 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:06:51,447 DEBUG Dhanu => PING(20) 2019-11-04 06:07:11,398 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:07:11,468 DEBUG Dhanu => PING(20) 2019-11-04 06:07:31,417 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:07:31,492 DEBUG Dhanu => PING(20) 2019-11-04 06:07:51,437 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:07:51,515 DEBUG Dhanu => PING(20) 2019-11-04 06:08:11,457 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:08:11,539 DEBUG Dhanu => PING(20) 2019-11-04 06:08:31,471 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:08:31,561 DEBUG Dhanu => PING(20) 2019-11-04 06:08:51,485 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:08:51,585 DEBUG Dhanu => PING(20) 2019-11-04 06:09:11,505 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:09:11,609 DEBUG Dhanu => PING(20) 2019-11-04 06:09:31,524 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:09:31,633 DEBUG Dhanu => PING(20) 2019-11-04 06:09:51,542 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:09:51,652 DEBUG Dhanu => PING(20) 2019-11-04 06:10:11,558 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:10:11,675 DEBUG Dhanu => PING(20) 2019-11-04 06:10:31,574 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:10:31,691 DEBUG Dhanu => PING(20) 2019-11-04 06:10:51,594 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:10:51,709 DEBUG Dhanu => PING(20) 2019-11-04 06:11:11,614 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:11:11,730 DEBUG Dhanu => PING(20) 2019-11-04 06:11:31,634 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:11:31,755 DEBUG Dhanu => PING(20) 2019-11-04 06:11:51,654 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:11:51,776 DEBUG Dhanu => PING(20) 2019-11-04 06:12:11,674 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:12:11,799 DEBUG Dhanu => PING(20) 2019-11-04 06:12:31,690 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:12:31,821 DEBUG Dhanu => PING(20) 2019-11-04 06:12:51,706 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:12:51,843 DEBUG Dhanu => PING(20) 2019-11-04 06:13:11,726 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:13:11,867 DEBUG Dhanu => PING(20) 2019-11-04 06:13:31,734 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:13:31,885 DEBUG Dhanu => PING(20) 2019-11-04 06:13:51,752 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:13:51,905 DEBUG Dhanu => PING(20) 2019-11-04 06:14:11,770 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:14:11,929 DEBUG Dhanu => PING(20) 2019-11-04 06:14:31,790 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:14:31,949 DEBUG Dhanu => PING(20) 2019-11-04 06:14:51,809 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:14:51,969 DEBUG Dhanu => PING(20) 2019-11-04 06:15:11,826 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:15:11,988 DEBUG Dhanu => PING(20) 2019-11-04 06:15:31,846 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:15:32,009 DEBUG Dhanu => PING(20) 2019-11-04 06:15:51,866 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:15:52,033 DEBUG Dhanu => PING(20) 2019-11-04 06:16:11,874 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:16:12,056 DEBUG Dhanu => PING(20) 2019-11-04 06:16:31,887 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:16:32,075 DEBUG Dhanu => PING(20) 2019-11-04 06:16:51,898 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:16:52,091 DEBUG Dhanu => PING(20) 2019-11-04 06:17:11,913 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:17:12,107 DEBUG Dhanu => PING(20) 2019-11-04 06:17:31,933 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:17:32,125 DEBUG Dhanu => PING(20) 2019-11-04 06:17:51,950 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:17:52,149 DEBUG Dhanu => PING(20) 2019-11-04 06:18:11,957 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:18:12,172 DEBUG Dhanu => PING(20) 2019-11-04 06:18:31,974 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:18:32,191 DEBUG Dhanu => PING(20) 2019-11-04 06:18:51,994 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:18:52,213 DEBUG Dhanu => PING(20) 2019-11-04 06:19:12,013 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:19:12,235 DEBUG Dhanu => PING(20) 2019-11-04 06:19:32,027 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:19:32,255 DEBUG Dhanu => PING(20) 2019-11-04 06:19:52,038 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:19:52,274 DEBUG Dhanu => PING(20) 2019-11-04 06:20:12,046 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:20:12,296 DEBUG Dhanu => PING(20) 2019-11-04 06:20:32,064 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:20:32,321 DEBUG Dhanu => PING(20) 2019-11-04 06:20:52,084 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:20:52,342 DEBUG Dhanu => PING(20) 2019-11-04 06:21:12,103 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:21:12,367 DEBUG Dhanu => PING(20) 2019-11-04 06:21:32,123 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:21:32,391 DEBUG Dhanu => PING(20) 2019-11-04 06:21:52,130 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:21:52,413 DEBUG Dhanu => PING(20) 2019-11-04 06:22:12,148 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:22:12,436 DEBUG Dhanu => PING(20) 2019-11-04 06:22:32,168 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:22:32,460 DEBUG Dhanu => PING(20) 2019-11-04 06:22:52,187 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:22:52,484 DEBUG Dhanu => PING(20) 2019-11-04 06:23:12,206 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:23:12,509 DEBUG Dhanu => PING(20) 2019-11-04 06:23:32,216 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:23:32,532 DEBUG Dhanu => PING(20) 2019-11-04 06:23:52,235 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:23:52,555 DEBUG Dhanu => PING(20) 2019-11-04 06:24:12,255 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:24:12,579 DEBUG Dhanu => PING(20) 2019-11-04 06:24:32,270 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:24:32,604 DEBUG Dhanu => PING(20) 2019-11-04 06:24:52,288 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:24:52,628 DEBUG Dhanu => PING(20) 2019-11-04 06:25:12,306 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:25:12,651 DEBUG Dhanu => PING(20) 2019-11-04 06:25:32,322 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:25:32,676 DEBUG Dhanu => PING(20) 2019-11-04 06:25:52,338 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:25:52,696 DEBUG Dhanu => PING(20) 2019-11-04 06:26:12,354 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:26:12,720 DEBUG Dhanu => PING(20) 2019-11-04 06:26:32,373 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:26:32,743 DEBUG Dhanu => PING(20) 2019-11-04 06:26:52,392 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:26:52,764 DEBUG Dhanu => PING(20) 2019-11-04 06:27:12,409 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:27:12,784 DEBUG Dhanu => PING(20) 2019-11-04 06:27:32,429 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:27:32,808 DEBUG Dhanu => PING(20) 2019-11-04 06:27:52,446 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:27:52,832 DEBUG Dhanu => PING(20) 2019-11-04 06:28:12,462 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:28:12,856 DEBUG Dhanu => PING(20) 2019-11-04 06:28:32,482 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:28:32,879 DEBUG Dhanu => PING(20) 2019-11-04 06:28:52,496 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:28:52,903 DEBUG Dhanu => PING(20) 2019-11-04 06:29:12,515 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:29:12,925 DEBUG Dhanu => PING(20) 2019-11-04 06:29:32,534 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:29:32,949 DEBUG Dhanu => PING(20) 2019-11-04 06:29:52,553 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:29:52,971 DEBUG Dhanu => PING(20) 2019-11-04 06:30:12,572 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:30:12,993 DEBUG Dhanu => PING(20) 2019-11-04 06:30:32,590 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:30:33,012 DEBUG Dhanu => PING(20) 2019-11-04 06:30:52,604 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:30:53,036 DEBUG Dhanu => PING(20) 2019-11-04 06:31:12,612 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:31:13,061 DEBUG Dhanu => PING(20) 2019-11-04 06:31:32,626 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:31:33,083 DEBUG Dhanu => PING(20) 2019-11-04 06:31:52,646 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:31:53,101 DEBUG Dhanu => PING(20) 2019-11-04 06:32:12,665 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:32:13,123 DEBUG Dhanu => PING(20) 2019-11-04 06:32:32,685 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:32:33,147 DEBUG Dhanu => PING(20) 2019-11-04 06:32:52,702 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:32:53,169 DEBUG Dhanu => PING(20) 2019-11-04 06:33:12,721 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:33:13,194 DEBUG Dhanu => PING(20) 2019-11-04 06:33:32,738 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:33:33,218 DEBUG Dhanu => PING(20) 2019-11-04 06:33:52,746 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:33:53,235 DEBUG Dhanu => PING(20) 2019-11-04 06:34:12,764 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:34:13,251 DEBUG Dhanu => PING(20) 2019-11-04 06:34:32,783 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:34:33,275 DEBUG Dhanu => PING(20) 2019-11-04 06:34:52,795 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:34:53,300 DEBUG Dhanu => PING(20) 2019-11-04 06:35:12,814 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:35:13,325 DEBUG Dhanu => PING(20) 2019-11-04 06:35:32,833 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:35:33,349 DEBUG Dhanu => PING(20) 2019-11-04 06:35:52,852 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:35:53,371 DEBUG Dhanu => PING(20) 2019-11-04 06:36:12,871 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:36:13,395 DEBUG Dhanu => PING(20) 2019-11-04 06:36:32,891 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:36:33,419 DEBUG Dhanu => PING(20) 2019-11-04 06:36:52,910 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:36:53,443 DEBUG Dhanu => PING(20) 2019-11-04 06:37:12,929 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:37:13,468 DEBUG Dhanu => PING(20) 2019-11-04 06:37:32,948 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:37:33,489 DEBUG Dhanu => PING(20) 2019-11-04 06:37:52,967 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:37:53,512 DEBUG Dhanu => PING(20) 2019-11-04 06:38:12,986 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:38:13,536 DEBUG Dhanu => PING(20) 2019-11-04 06:38:33,005 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:38:33,559 DEBUG Dhanu => PING(20) 2019-11-04 06:38:53,022 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:38:53,583 DEBUG Dhanu => PING(20) 2019-11-04 06:39:13,039 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:39:13,607 DEBUG Dhanu => PING(20) 2019-11-04 06:39:33,059 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:39:33,623 DEBUG Dhanu => PING(20) 2019-11-04 06:39:53,078 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:39:53,648 DEBUG Dhanu => PING(20) 2019-11-04 06:40:13,089 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:40:13,673 DEBUG Dhanu => PING(20) 2019-11-04 06:40:33,108 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:40:33,697 DEBUG Dhanu => PING(20) 2019-11-04 06:40:53,128 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:40:53,722 DEBUG Dhanu => PING(20) 2019-11-04 06:41:13,147 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:41:13,743 DEBUG Dhanu => PING(20) 2019-11-04 06:41:33,166 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:41:33,768 DEBUG Dhanu => PING(20) 2019-11-04 06:41:53,184 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:41:53,789 DEBUG Dhanu => PING(20) 2019-11-04 06:42:13,204 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:42:13,813 DEBUG Dhanu => PING(20) 2019-11-04 06:42:33,223 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:42:33,832 DEBUG Dhanu => PING(20) 2019-11-04 06:42:53,242 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:42:53,855 DEBUG Dhanu => PING(20) 2019-11-04 06:43:13,255 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:43:13,879 DEBUG Dhanu => PING(20) 2019-11-04 06:43:33,263 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:43:33,904 DEBUG Dhanu => PING(20) 2019-11-04 06:43:53,271 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:43:53,928 DEBUG Dhanu => PING(20) 2019-11-04 06:44:13,278 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:44:13,953 DEBUG Dhanu => PING(20) 2019-11-04 06:44:33,285 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:44:33,977 DEBUG Dhanu => PING(20) 2019-11-04 06:44:53,300 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:44:53,998 DEBUG Dhanu => PING(20) 2019-11-04 06:45:13,318 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:45:14,019 DEBUG Dhanu => PING(20) 2019-11-04 06:45:33,336 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:45:34,043 DEBUG Dhanu => PING(20) 2019-11-04 06:45:53,355 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:45:54,067 DEBUG Dhanu => PING(20) 2019-11-04 06:46:13,374 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:46:14,092 DEBUG Dhanu => PING(20) 2019-11-04 06:46:33,388 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:46:34,116 DEBUG Dhanu => PING(20) 2019-11-04 06:46:53,404 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:46:54,141 DEBUG Dhanu => PING(20) 2019-11-04 06:47:13,423 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:47:14,165 DEBUG Dhanu => PING(20) 2019-11-04 06:47:33,442 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:47:34,185 DEBUG Dhanu => PING(20) 2019-11-04 06:47:53,461 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:47:54,209 DEBUG Dhanu => PING(20) 2019-11-04 06:48:13,480 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:48:14,228 DEBUG Dhanu => PING(20) 2019-11-04 06:48:33,492 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:48:34,248 DEBUG Dhanu => PING(20) 2019-11-04 06:48:53,512 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:48:54,268 DEBUG Dhanu => PING(20) 2019-11-04 06:49:13,520 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:49:14,289 DEBUG Dhanu => PING(20) 2019-11-04 06:49:33,527 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:49:34,314 DEBUG Dhanu => PING(20) 2019-11-04 06:49:53,533 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:49:54,335 DEBUG Dhanu => PING(20) 2019-11-04 06:50:13,542 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:50:14,359 DEBUG Dhanu => PING(20) 2019-11-04 06:50:33,549 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:50:34,383 DEBUG Dhanu => PING(20) 2019-11-04 06:50:53,556 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:50:54,404 DEBUG Dhanu => PING(20) 2019-11-04 06:51:13,564 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:51:14,429 DEBUG Dhanu => PING(20) 2019-11-04 06:51:33,572 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:51:34,453 DEBUG Dhanu => PING(20) 2019-11-04 06:51:53,582 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:51:54,472 DEBUG Dhanu => PING(20) 2019-11-04 06:52:13,598 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:52:14,495 DEBUG Dhanu => PING(20) 2019-11-04 06:52:33,617 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:52:34,520 DEBUG Dhanu => PING(20) 2019-11-04 06:52:53,636 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:52:54,544 DEBUG Dhanu => PING(20) 2019-11-04 06:53:13,647 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:53:14,567 DEBUG Dhanu => PING(20) 2019-11-04 06:53:33,663 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:53:34,591 DEBUG Dhanu => PING(20) 2019-11-04 06:53:53,680 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:53:54,611 DEBUG Dhanu => PING(20) 2019-11-04 06:54:13,693 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:54:14,630 DEBUG Dhanu => PING(20) 2019-11-04 06:54:33,701 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:54:34,655 DEBUG Dhanu => PING(20) 2019-11-04 06:54:53,718 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:54:54,679 DEBUG Dhanu => PING(20) 2019-11-04 06:55:13,737 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:55:14,697 DEBUG Dhanu => PING(20) 2019-11-04 06:55:33,754 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:55:34,721 DEBUG Dhanu => PING(20) 2019-11-04 06:55:53,773 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:55:54,745 DEBUG Dhanu => PING(20) 2019-11-04 06:56:13,781 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:56:14,769 DEBUG Dhanu => PING(20) 2019-11-04 06:56:33,797 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:56:34,792 DEBUG Dhanu => PING(20) 2019-11-04 06:56:53,814 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:56:54,809 DEBUG Dhanu => PING(20) 2019-11-04 06:57:13,830 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:57:14,831 DEBUG Dhanu => PING(20) 2019-11-04 06:57:33,849 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:57:34,855 DEBUG Dhanu => PING(20) 2019-11-04 06:57:53,868 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:57:54,880 DEBUG Dhanu => PING(20) 2019-11-04 06:58:13,887 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:58:14,904 DEBUG Dhanu => PING(20) 2019-11-04 06:58:33,905 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:58:34,929 DEBUG Dhanu => PING(20) 2019-11-04 06:58:53,924 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:58:54,953 DEBUG Dhanu => PING(20) 2019-11-04 06:59:13,931 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:59:14,971 DEBUG Dhanu => PING(20) 2019-11-04 06:59:33,945 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:59:34,996 DEBUG Dhanu => PING(20) 2019-11-04 06:59:53,954 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 06:59:55,017 DEBUG Dhanu => PING(20) 2019-11-04 07:00:13,973 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:00:15,039 DEBUG Dhanu => PING(20) 2019-11-04 07:00:33,991 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:00:35,060 DEBUG Dhanu => PING(20) 2019-11-04 07:00:54,010 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:00:55,084 DEBUG Dhanu => PING(20) 2019-11-04 07:01:14,024 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:01:15,107 DEBUG Dhanu => PING(20) 2019-11-04 07:01:34,042 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:01:35,132 DEBUG Dhanu => PING(20) 2019-11-04 07:01:54,061 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:01:55,155 DEBUG Dhanu => PING(20) 2019-11-04 07:02:14,079 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:02:15,180 DEBUG Dhanu => PING(20) 2019-11-04 07:02:34,098 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:02:35,200 DEBUG Dhanu => PING(20) 2019-11-04 07:02:54,106 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:02:55,219 DEBUG Dhanu => PING(20) 2019-11-04 07:03:14,122 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:03:15,240 DEBUG Dhanu => PING(20) 2019-11-04 07:03:34,130 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:03:35,265 DEBUG Dhanu => PING(20) 2019-11-04 07:03:54,143 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:03:55,289 DEBUG Dhanu => PING(20) 2019-11-04 07:04:14,156 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:04:15,307 DEBUG Dhanu => PING(20) 2019-11-04 07:04:34,174 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:04:35,327 DEBUG Dhanu => PING(20) 2019-11-04 07:04:54,193 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:04:55,347 DEBUG Dhanu => PING(20) 2019-11-04 07:05:14,210 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:05:15,368 DEBUG Dhanu => PING(20) 2019-11-04 07:05:34,229 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:05:35,391 DEBUG Dhanu => PING(20) 2019-11-04 07:05:54,242 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:05:55,415 DEBUG Dhanu => PING(20) 2019-11-04 07:06:14,261 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:06:15,439 DEBUG Dhanu => PING(20) 2019-11-04 07:06:34,269 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:06:35,463 DEBUG Dhanu => PING(20) 2019-11-04 07:06:54,281 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:06:55,488 DEBUG Dhanu => PING(20) 2019-11-04 07:07:14,289 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:07:15,512 DEBUG Dhanu => PING(20) 2019-11-04 07:07:34,297 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:07:35,533 DEBUG Dhanu => PING(20) 2019-11-04 07:07:54,305 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:07:55,557 DEBUG Dhanu => PING(20) 2019-11-04 07:08:14,317 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:08:15,574 DEBUG Dhanu => PING(20) 2019-11-04 07:08:34,335 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:08:35,597 DEBUG Dhanu => PING(20) 2019-11-04 07:08:54,353 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:08:55,619 DEBUG Dhanu => PING(20) 2019-11-04 07:09:14,372 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:09:15,639 DEBUG Dhanu => PING(20) 2019-11-04 07:09:34,390 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:09:35,657 DEBUG Dhanu => PING(20) 2019-11-04 07:09:54,409 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:09:55,681 DEBUG Dhanu => PING(20) 2019-11-04 07:10:14,417 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:10:15,699 DEBUG Dhanu => PING(20) 2019-11-04 07:10:34,432 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:10:35,719 DEBUG Dhanu => PING(20) 2019-11-04 07:10:54,440 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:10:55,743 DEBUG Dhanu => PING(20) 2019-11-04 07:11:14,457 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:11:15,762 DEBUG Dhanu => PING(20) 2019-11-04 07:11:34,475 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:11:35,783 DEBUG Dhanu => PING(20) 2019-11-04 07:11:54,483 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:11:55,809 DEBUG Dhanu => PING(20) 2019-11-04 07:12:14,496 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:12:15,833 DEBUG Dhanu => PING(20) 2019-11-04 07:12:34,514 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:12:35,853 DEBUG Dhanu => PING(20) 2019-11-04 07:12:54,523 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:12:55,877 DEBUG Dhanu => PING(20) 2019-11-04 07:13:14,540 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:13:15,900 DEBUG Dhanu => PING(20) 2019-11-04 07:13:34,559 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:13:35,920 DEBUG Dhanu => PING(20) 2019-11-04 07:13:54,577 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:13:55,944 DEBUG Dhanu => PING(20) 2019-11-04 07:14:14,595 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:14:15,969 DEBUG Dhanu => PING(20) 2019-11-04 07:14:34,613 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:14:35,993 DEBUG Dhanu => PING(20) 2019-11-04 07:14:54,631 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:14:56,017 DEBUG Dhanu => PING(20) 2019-11-04 07:15:14,640 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:15:16,036 DEBUG Dhanu => PING(20) 2019-11-04 07:15:34,657 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:15:36,057 DEBUG Dhanu => PING(20) 2019-11-04 07:15:54,675 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:15:56,082 DEBUG Dhanu => PING(20) 2019-11-04 07:16:14,690 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:16:16,106 DEBUG Dhanu => PING(20) 2019-11-04 07:16:34,709 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:16:36,131 DEBUG Dhanu => PING(20) 2019-11-04 07:16:54,726 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:16:56,147 DEBUG Dhanu => PING(20) 2019-11-04 07:17:14,742 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:17:16,167 DEBUG Dhanu => PING(20) 2019-11-04 07:17:34,758 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:17:36,184 DEBUG Dhanu => PING(20) 2019-11-04 07:17:54,774 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:17:56,199 DEBUG Dhanu => PING(20) 2019-11-04 07:18:14,792 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:18:16,215 DEBUG Dhanu => PING(20) 2019-11-04 07:18:34,811 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:18:36,231 DEBUG Dhanu => PING(20) 2019-11-04 07:18:54,829 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:18:56,251 DEBUG Dhanu => PING(20) 2019-11-04 07:19:14,847 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:19:16,273 DEBUG Dhanu => PING(20) 2019-11-04 07:19:34,863 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:19:36,295 DEBUG Dhanu => PING(20) 2019-11-04 07:19:54,878 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:19:56,311 DEBUG Dhanu => PING(20) 2019-11-04 07:20:14,894 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:20:16,331 DEBUG Dhanu => PING(20) 2019-11-04 07:20:34,910 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:20:36,347 DEBUG Dhanu => PING(20) 2019-11-04 07:20:54,918 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:20:56,366 DEBUG Dhanu => PING(20) 2019-11-04 07:21:14,929 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:21:16,389 DEBUG Dhanu => PING(20) 2019-11-04 07:21:34,948 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:21:36,412 DEBUG Dhanu => PING(20) 2019-11-04 07:21:54,966 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:21:56,435 DEBUG Dhanu => PING(20) 2019-11-04 07:22:14,982 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:22:16,453 DEBUG Dhanu => PING(20) 2019-11-04 07:22:34,990 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:22:36,474 DEBUG Dhanu => PING(20) 2019-11-04 07:22:55,008 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:22:56,495 DEBUG Dhanu => PING(20) 2019-11-04 07:23:15,026 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:23:16,515 DEBUG Dhanu => PING(20) 2019-11-04 07:23:35,044 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:23:36,536 DEBUG Dhanu => PING(20) 2019-11-04 07:23:55,062 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:23:56,559 DEBUG Dhanu => PING(20) 2019-11-04 07:24:15,080 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:24:16,584 DEBUG Dhanu => PING(20) 2019-11-04 07:24:35,098 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:24:36,603 DEBUG Dhanu => PING(20) 2019-11-04 07:24:55,106 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:24:56,619 DEBUG Dhanu => PING(20) 2019-11-04 07:25:15,123 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:25:16,642 DEBUG Dhanu => PING(20) 2019-11-04 07:25:35,139 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:25:36,659 DEBUG Dhanu => PING(20) 2019-11-04 07:25:55,155 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:25:56,677 DEBUG Dhanu => PING(20) 2019-11-04 07:26:15,171 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:26:16,696 DEBUG Dhanu => PING(20) 2019-11-04 07:26:35,186 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:26:36,714 DEBUG Dhanu => PING(20) 2019-11-04 07:26:55,202 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:26:56,735 DEBUG Dhanu => PING(20) 2019-11-04 07:27:15,220 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:27:16,756 DEBUG Dhanu => PING(20) 2019-11-04 07:27:35,239 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:27:36,780 DEBUG Dhanu => PING(20) 2019-11-04 07:27:55,257 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:27:56,798 DEBUG Dhanu => PING(20) 2019-11-04 07:28:15,275 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:28:16,819 DEBUG Dhanu => PING(20) 2019-11-04 07:28:35,293 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:28:36,839 DEBUG Dhanu => PING(20) 2019-11-04 07:28:55,311 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:28:56,863 DEBUG Dhanu => PING(20) 2019-11-04 07:29:15,329 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:29:16,879 DEBUG Dhanu => PING(20) 2019-11-04 07:29:35,347 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:29:36,903 DEBUG Dhanu => PING(20) 2019-11-04 07:29:55,360 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:29:56,927 DEBUG Dhanu => PING(20) 2019-11-04 07:30:15,378 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:30:16,948 DEBUG Dhanu => PING(20) 2019-11-04 07:30:35,396 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:30:36,972 DEBUG Dhanu => PING(20) 2019-11-04 07:30:55,415 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:30:56,997 DEBUG Dhanu => PING(20) 2019-11-04 07:31:15,426 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:31:17,019 DEBUG Dhanu => PING(20) 2019-11-04 07:31:35,441 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:31:37,038 DEBUG Dhanu => PING(20) 2019-11-04 07:31:55,459 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:31:57,060 DEBUG Dhanu => PING(20) 2019-11-04 07:32:15,470 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:32:17,085 DEBUG Dhanu => PING(20) 2019-11-04 07:32:35,486 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:32:37,109 DEBUG Dhanu => PING(20) 2019-11-04 07:32:55,494 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:32:57,129 DEBUG Dhanu => PING(20) 2019-11-04 07:33:15,504 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:33:17,153 DEBUG Dhanu => PING(20) 2019-11-04 07:33:35,521 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:33:37,176 DEBUG Dhanu => PING(20) 2019-11-04 07:33:55,539 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:33:57,201 DEBUG Dhanu => PING(20) 2019-11-04 07:34:15,547 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:34:17,224 DEBUG Dhanu => PING(20) 2019-11-04 07:34:35,559 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:34:37,248 DEBUG Dhanu => PING(20) 2019-11-04 07:34:55,577 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:34:57,273 DEBUG Dhanu => PING(20) 2019-11-04 07:35:15,595 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:35:17,297 DEBUG Dhanu => PING(20) 2019-11-04 07:35:35,613 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:35:37,319 DEBUG Dhanu => PING(20) 2019-11-04 07:35:55,631 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:35:57,340 DEBUG Dhanu => PING(20) 2019-11-04 07:36:15,649 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:36:17,361 DEBUG Dhanu => PING(20) 2019-11-04 07:36:35,667 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:36:37,383 DEBUG Dhanu => PING(20) 2019-11-04 07:36:55,685 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:36:57,406 DEBUG Dhanu => PING(20) 2019-11-04 07:37:15,703 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:37:17,425 DEBUG Dhanu => PING(20) 2019-11-04 07:37:35,710 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:37:37,447 DEBUG Dhanu => PING(20) 2019-11-04 07:37:55,718 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:37:57,468 DEBUG Dhanu => PING(20) 2019-11-04 07:38:15,732 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:38:17,493 DEBUG Dhanu => PING(20) 2019-11-04 07:38:35,750 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:38:37,515 DEBUG Dhanu => PING(20) 2019-11-04 07:38:55,758 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:38:57,537 DEBUG Dhanu => PING(20) 2019-11-04 07:39:15,774 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:39:17,561 DEBUG Dhanu => PING(20) 2019-11-04 07:39:35,792 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:39:37,581 DEBUG Dhanu => PING(20) 2019-11-04 07:39:55,810 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:39:57,605 DEBUG Dhanu => PING(20) 2019-11-04 07:40:15,828 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:40:17,629 DEBUG Dhanu => PING(20) 2019-11-04 07:40:35,846 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:40:37,653 DEBUG Dhanu => PING(20) 2019-11-04 07:40:55,864 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:40:57,675 DEBUG Dhanu => PING(20) 2019-11-04 07:41:15,882 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:41:17,700 DEBUG Dhanu => PING(20) 2019-11-04 07:41:35,901 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:41:37,723 DEBUG Dhanu => PING(20) 2019-11-04 07:41:55,919 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:41:57,741 DEBUG Dhanu => PING(20) 2019-11-04 07:42:15,937 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:42:17,765 DEBUG Dhanu => PING(20) 2019-11-04 07:42:35,955 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:42:37,789 DEBUG Dhanu => PING(20) 2019-11-04 07:42:55,963 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:42:57,806 DEBUG Dhanu => PING(20) 2019-11-04 07:43:15,977 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:43:17,823 DEBUG Dhanu => PING(20) 2019-11-04 07:43:35,995 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:43:37,847 DEBUG Dhanu => PING(20) 2019-11-04 07:43:56,010 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:43:57,868 DEBUG Dhanu => PING(20) 2019-11-04 07:44:16,028 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:44:17,891 DEBUG Dhanu => PING(20) 2019-11-04 07:44:36,036 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:44:37,916 DEBUG Dhanu => PING(20) 2019-11-04 07:44:56,052 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:44:57,939 DEBUG Dhanu => PING(20) 2019-11-04 07:45:16,067 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:45:17,959 DEBUG Dhanu => PING(20) 2019-11-04 07:45:36,084 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:45:37,980 DEBUG Dhanu => PING(20) 2019-11-04 07:45:56,102 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:45:58,005 DEBUG Dhanu => PING(20) 2019-11-04 07:46:16,120 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:46:18,023 DEBUG Dhanu => PING(20) 2019-11-04 07:46:36,138 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:46:38,039 DEBUG Dhanu => PING(20) 2019-11-04 07:46:56,156 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:46:58,063 DEBUG Dhanu => PING(20) 2019-11-04 07:47:16,173 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 07:47:18,088 DEBUG Dhanu => PING(20) 2019-11-04 08:09:59,127 DEBUG Dhanu => PING(20) 2019-11-04 08:09:59,186 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:10:19,144 DEBUG Dhanu => PING(20) 2019-11-04 08:10:19,190 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:10:39,164 DEBUG Dhanu => PING(20) 2019-11-04 08:10:39,195 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:10:59,179 DEBUG Dhanu => PING(20) 2019-11-04 08:10:59,203 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:11:19,196 DEBUG Dhanu => PING(20) 2019-11-04 08:11:19,208 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:11:39,212 DEBUG Dhanu => PING(20) 2019-11-04 08:11:39,262 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:11:59,228 DEBUG Dhanu => PING(20) 2019-11-04 08:11:59,267 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:12:19,243 DEBUG Dhanu => PING(20) 2019-11-04 08:12:19,273 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:12:39,261 DEBUG Dhanu => PING(20) 2019-11-04 08:12:39,301 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:12:59,279 DEBUG Dhanu => PING(20) 2019-11-04 08:12:59,308 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:13:19,295 DEBUG Dhanu => PING(20) 2019-11-04 08:13:19,314 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:13:39,311 DEBUG Dhanu => PING(20) 2019-11-04 08:13:39,329 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:13:59,327 DEBUG Dhanu => PING(20) 2019-11-04 08:13:59,353 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:14:19,348 DEBUG Dhanu => PING(20) 2019-11-04 08:14:19,377 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:14:39,361 DEBUG Dhanu => PING(20) 2019-11-04 08:14:39,391 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:14:59,383 DEBUG Dhanu => PING(20) 2019-11-04 08:14:59,414 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:15:19,404 DEBUG Dhanu => PING(20) 2019-11-04 08:15:19,427 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:15:39,424 DEBUG Dhanu => PING(20) 2019-11-04 08:15:39,451 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:15:42,343 INFO scheduling health checks: 2019-11-04 08:15:43,563 INFO scheduling jobs: 2019-11-04 08:15:43,621 DEBUG - kvm 2019-11-04 08:15:50,017 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 08:15:50,018 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 799, in main_loop jobs = schedule(self.logger, self.events["available_dt"]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 08:15:50,360 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 08:16:04,939 INFO [INIT] Dropping privileges 2019-11-04 08:16:04,939 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 08:16:04,939 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 08:16:04,941 INFO [INIT] Marking all workers as offline 2019-11-04 08:16:04,999 INFO [INIT] Starting encryption 2019-11-04 08:16:05,001 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 08:16:05,001 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 08:16:05,002 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 08:16:05,016 INFO [INIT] LAVA master has started. 2019-11-04 08:16:05,017 INFO [INIT] Using protocol version 3 2019-11-04 08:16:05,045 DEBUG Dhanu => PING(20) 2019-11-04 08:16:05,045 WARNING Unknown dispatcher (server crashed) 2019-11-04 08:16:25,035 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:16:25,064 DEBUG Dhanu => PING(20) 2019-11-04 08:16:45,051 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:16:45,081 DEBUG Dhanu => PING(20) 2019-11-04 08:17:05,070 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:17:05,105 DEBUG Dhanu => PING(20) 2019-11-04 08:17:25,087 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:17:25,129 DEBUG Dhanu => PING(20) 2019-11-04 08:17:45,107 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:17:45,153 DEBUG Dhanu => PING(20) 2019-11-04 08:18:05,126 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:18:05,176 DEBUG Dhanu => PING(20) 2019-11-04 08:18:25,140 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:18:25,198 DEBUG Dhanu => PING(20) 2019-11-04 08:18:45,160 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:18:45,222 DEBUG Dhanu => PING(20) 2019-11-04 08:19:05,180 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:19:05,243 DEBUG Dhanu => PING(20) 2019-11-04 08:19:25,200 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:19:25,264 DEBUG Dhanu => PING(20) 2019-11-04 08:19:45,219 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:19:45,288 DEBUG Dhanu => PING(20) 2019-11-04 08:20:05,238 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:20:05,311 DEBUG Dhanu => PING(20) 2019-11-04 08:20:25,246 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:20:25,335 DEBUG Dhanu => PING(20) 2019-11-04 08:20:45,259 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:20:45,360 DEBUG Dhanu => PING(20) 2019-11-04 08:21:05,279 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:21:05,377 DEBUG Dhanu => PING(20) 2019-11-04 08:21:25,299 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:21:25,398 DEBUG Dhanu => PING(20) 2019-11-04 08:21:45,319 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:21:45,418 DEBUG Dhanu => PING(20) 2019-11-04 08:22:05,339 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:22:05,442 DEBUG Dhanu => PING(20) 2019-11-04 08:22:25,350 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:22:25,467 DEBUG Dhanu => PING(20) 2019-11-04 08:22:45,369 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:22:45,492 DEBUG Dhanu => PING(20) 2019-11-04 08:23:05,377 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:23:05,516 DEBUG Dhanu => PING(20) 2019-11-04 08:23:25,389 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:23:25,540 DEBUG Dhanu => PING(20) 2019-11-04 08:23:45,409 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:23:45,559 DEBUG Dhanu => PING(20) 2019-11-04 08:24:05,429 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:24:05,578 DEBUG Dhanu => PING(20) 2019-11-04 08:24:25,449 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:24:25,600 DEBUG Dhanu => PING(20) 2019-11-04 08:24:45,467 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:24:45,624 DEBUG Dhanu => PING(20) 2019-11-04 08:25:05,486 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:25:05,648 DEBUG Dhanu => PING(20) 2019-11-04 08:25:25,506 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:25:25,673 DEBUG Dhanu => PING(20) 2019-11-04 08:25:45,523 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:25:45,697 DEBUG Dhanu => PING(20) 2019-11-04 08:26:05,542 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:26:05,720 DEBUG Dhanu => PING(20) 2019-11-04 08:26:25,557 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:26:25,745 DEBUG Dhanu => PING(20) 2019-11-04 08:26:45,577 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:26:45,764 DEBUG Dhanu => PING(20) 2019-11-04 08:27:05,596 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:27:05,784 DEBUG Dhanu => PING(20) 2019-11-04 08:27:25,616 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:27:25,808 DEBUG Dhanu => PING(20) 2019-11-04 08:27:45,635 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:27:45,829 DEBUG Dhanu => PING(20) 2019-11-04 08:28:05,654 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:28:05,853 DEBUG Dhanu => PING(20) 2019-11-04 08:28:25,673 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:28:25,878 DEBUG Dhanu => PING(20) 2019-11-04 08:28:45,691 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:28:45,902 DEBUG Dhanu => PING(20) 2019-11-04 08:29:05,711 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:29:05,924 DEBUG Dhanu => PING(20) 2019-11-04 08:29:18,459 INFO [POLL] Received a signal, leaving 2019-11-04 08:29:18,460 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 08:29:19,431 INFO [INIT] Dropping privileges 2019-11-04 08:29:19,431 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 08:29:19,431 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 08:29:19,433 INFO [INIT] Marking all workers as offline 2019-11-04 08:29:19,508 INFO [INIT] Starting encryption 2019-11-04 08:29:19,510 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 08:29:19,510 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 08:29:19,511 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 08:29:19,526 INFO [INIT] LAVA master has started. 2019-11-04 08:29:19,526 INFO [INIT] Using protocol version 3 2019-11-04 08:29:25,944 DEBUG Dhanu => PING(20) 2019-11-04 08:29:25,944 WARNING Unknown dispatcher (server crashed) 2019-11-04 08:29:26,994 INFO Dhanu => HELLO 2019-11-04 08:29:26,994 WARNING Dispatcher has RESTARTED 2019-11-04 08:29:32,001 DEBUG Dhanu => PING(20) 2019-11-04 08:29:39,531 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:29:52,026 DEBUG Dhanu => PING(20) 2019-11-04 08:29:59,538 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:30:12,046 DEBUG Dhanu => PING(20) 2019-11-04 08:30:19,545 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:30:32,063 DEBUG Dhanu => PING(20) 2019-11-04 08:30:39,553 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:30:52,080 DEBUG Dhanu => PING(20) 2019-11-04 08:30:53,102 INFO [POLL] Received a signal, leaving 2019-11-04 08:30:53,103 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:05:37,415 INFO [INIT] Dropping privileges 2019-11-04 14:05:37,479 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:05:37,480 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:05:37,557 INFO [INIT] Marking all workers as offline 2019-11-04 14:05:37,688 INFO [INIT] Starting encryption 2019-11-04 14:05:37,690 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:05:37,732 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 14:05:37,733 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 14:05:37,740 INFO [INIT] LAVA master has started. 2019-11-04 14:05:37,740 INFO [INIT] Using protocol version 3 2019-11-04 14:05:37,829 INFO Dhanu => HELLO 2019-11-04 14:05:37,830 WARNING New dispatcher 2019-11-04 14:05:37,909 INFO Dhanu => HELLO_RETRY 2019-11-04 14:05:37,910 WARNING Dispatcher was not confirmed 2019-11-04 14:05:37,954 INFO Dhanu => HELLO_RETRY 2019-11-04 14:05:37,955 WARNING Dispatcher was not confirmed 2019-11-04 14:05:37,998 INFO Dhanu => HELLO_RETRY 2019-11-04 14:05:37,999 WARNING Dispatcher was not confirmed 2019-11-04 14:05:38,043 INFO Dhanu => HELLO_RETRY 2019-11-04 14:05:38,043 WARNING Dispatcher was not confirmed 2019-11-04 14:05:38,087 INFO Dhanu => HELLO_RETRY 2019-11-04 14:05:38,088 WARNING Dispatcher was not confirmed 2019-11-04 14:05:43,095 DEBUG Dhanu => PING(20) 2019-11-04 14:05:57,752 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:06:03,118 DEBUG Dhanu => PING(20) 2019-11-04 14:06:18,077 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:06:23,136 DEBUG Dhanu => PING(20) 2019-11-04 14:06:38,086 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:06:43,156 DEBUG Dhanu => PING(20) 2019-11-04 14:06:58,093 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:07:03,176 DEBUG Dhanu => PING(20) 2019-11-04 14:07:18,100 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:07:23,192 DEBUG Dhanu => PING(20) 2019-11-04 14:07:38,107 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:07:43,209 DEBUG Dhanu => PING(20) 2019-11-04 14:07:58,111 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:08:03,225 DEBUG Dhanu => PING(20) 2019-11-04 14:08:18,117 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:08:23,241 DEBUG Dhanu => PING(20) 2019-11-04 14:08:38,124 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:08:43,265 DEBUG Dhanu => PING(20) 2019-11-04 14:08:58,132 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:09:03,281 DEBUG Dhanu => PING(20) 2019-11-04 14:09:18,140 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:09:23,316 DEBUG Dhanu => PING(20) 2019-11-04 14:09:38,147 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:09:43,336 DEBUG Dhanu => PING(20) 2019-11-04 14:09:58,155 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:10:03,356 DEBUG Dhanu => PING(20) 2019-11-04 14:10:18,163 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:10:23,381 DEBUG Dhanu => PING(20) 2019-11-04 14:10:38,170 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:10:43,402 DEBUG Dhanu => PING(20) 2019-11-04 14:10:58,176 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:11:03,421 DEBUG Dhanu => PING(20) 2019-11-04 14:11:18,184 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:11:23,442 DEBUG Dhanu => PING(20) 2019-11-04 14:11:38,192 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:11:43,462 DEBUG Dhanu => PING(20) 2019-11-04 14:11:58,199 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:12:03,481 DEBUG Dhanu => PING(20) 2019-11-04 14:12:18,208 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:12:23,502 DEBUG Dhanu => PING(20) 2019-11-04 14:12:38,216 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:12:43,524 DEBUG Dhanu => PING(20) 2019-11-04 14:12:58,223 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:13:03,540 DEBUG Dhanu => PING(20) 2019-11-04 14:13:18,227 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:13:23,558 DEBUG Dhanu => PING(20) 2019-11-04 14:13:38,235 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:13:43,573 DEBUG Dhanu => PING(20) 2019-11-04 14:13:58,242 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:14:03,594 DEBUG Dhanu => PING(20) 2019-11-04 14:14:18,252 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:14:23,613 DEBUG Dhanu => PING(20) 2019-11-04 14:14:38,259 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:14:43,632 DEBUG Dhanu => PING(20) 2019-11-04 14:14:58,266 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:15:03,652 DEBUG Dhanu => PING(20) 2019-11-04 14:15:18,273 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:15:23,673 DEBUG Dhanu => PING(20) 2019-11-04 14:15:38,282 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:15:43,694 DEBUG Dhanu => PING(20) 2019-11-04 14:15:58,289 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 14:16:03,714 DEBUG Dhanu => PING(20) 2019-11-04 14:16:18,296 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:45:28,169 INFO [POLL] Received a signal, leaving 2019-11-04 08:45:28,169 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 08:45:29,138 INFO [INIT] Dropping privileges 2019-11-04 08:45:29,138 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 08:45:29,138 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 08:45:29,140 INFO [INIT] Marking all workers as offline 2019-11-04 08:45:29,164 INFO [INIT] Starting encryption 2019-11-04 08:45:29,165 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 08:45:29,165 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 08:45:29,166 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 08:45:29,179 INFO [INIT] LAVA master has started. 2019-11-04 08:45:29,179 INFO [INIT] Using protocol version 3 2019-11-04 08:45:33,537 INFO Dhanu => HELLO 2019-11-04 08:45:33,538 WARNING New dispatcher 2019-11-04 08:45:38,544 DEBUG Dhanu => PING(20) 2019-11-04 08:45:49,188 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:45:58,565 DEBUG Dhanu => PING(20) 2019-11-04 08:46:09,202 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:46:18,583 DEBUG Dhanu => PING(20) 2019-11-04 08:46:29,209 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:46:38,603 DEBUG Dhanu => PING(20) 2019-11-04 08:46:49,217 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:46:58,623 DEBUG Dhanu => PING(20) 2019-11-04 08:47:09,224 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:47:18,643 DEBUG Dhanu => PING(20) 2019-11-04 08:47:29,234 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:47:38,665 DEBUG Dhanu => PING(20) 2019-11-04 08:47:49,245 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:47:58,686 DEBUG Dhanu => PING(20) 2019-11-04 08:48:09,255 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:48:18,709 DEBUG Dhanu => PING(20) 2019-11-04 08:48:29,266 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:48:38,731 DEBUG Dhanu => PING(20) 2019-11-04 08:48:49,274 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:48:58,751 DEBUG Dhanu => PING(20) 2019-11-04 08:49:09,280 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:49:18,770 DEBUG Dhanu => PING(20) 2019-11-04 08:49:29,288 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:49:38,785 DEBUG Dhanu => PING(20) 2019-11-04 08:49:49,296 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:49:58,801 DEBUG Dhanu => PING(20) 2019-11-04 08:50:09,304 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:50:18,818 DEBUG Dhanu => PING(20) 2019-11-04 08:50:29,314 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:50:38,842 DEBUG Dhanu => PING(20) 2019-11-04 08:50:49,324 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:50:58,865 DEBUG Dhanu => PING(20) 2019-11-04 08:51:09,333 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:51:18,886 DEBUG Dhanu => PING(20) 2019-11-04 08:51:29,340 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:51:38,907 DEBUG Dhanu => PING(20) 2019-11-04 08:51:49,348 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:51:58,926 DEBUG Dhanu => PING(20) 2019-11-04 08:52:09,356 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:52:18,943 DEBUG Dhanu => PING(20) 2019-11-04 08:52:29,360 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:52:38,968 DEBUG Dhanu => PING(20) 2019-11-04 08:52:49,364 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:52:58,989 DEBUG Dhanu => PING(20) 2019-11-04 08:53:09,367 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:53:19,005 DEBUG Dhanu => PING(20) 2019-11-04 08:53:29,374 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:53:39,027 DEBUG Dhanu => PING(20) 2019-11-04 08:53:49,382 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:53:59,049 DEBUG Dhanu => PING(20) 2019-11-04 08:54:09,390 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:54:19,071 DEBUG Dhanu => PING(20) 2019-11-04 08:54:29,401 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:54:39,096 DEBUG Dhanu => PING(20) 2019-11-04 08:54:49,411 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:54:59,120 DEBUG Dhanu => PING(20) 2019-11-04 08:55:09,421 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:55:19,133 DEBUG Dhanu => PING(20) 2019-11-04 08:55:29,428 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:55:39,159 DEBUG Dhanu => PING(20) 2019-11-04 08:55:49,438 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:55:59,181 DEBUG Dhanu => PING(20) 2019-11-04 08:56:09,447 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:56:19,206 DEBUG Dhanu => PING(20) 2019-11-04 08:56:29,456 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:56:39,227 DEBUG Dhanu => PING(20) 2019-11-04 08:56:49,466 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:56:59,252 DEBUG Dhanu => PING(20) 2019-11-04 08:57:09,476 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:57:19,273 DEBUG Dhanu => PING(20) 2019-11-04 08:57:29,487 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:57:39,297 DEBUG Dhanu => PING(20) 2019-11-04 08:57:49,497 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:57:59,319 DEBUG Dhanu => PING(20) 2019-11-04 08:58:09,507 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:58:19,341 DEBUG Dhanu => PING(20) 2019-11-04 08:58:29,514 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:58:39,363 DEBUG Dhanu => PING(20) 2019-11-04 08:58:49,522 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:58:59,387 DEBUG Dhanu => PING(20) 2019-11-04 08:59:09,528 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:59:19,404 DEBUG Dhanu => PING(20) 2019-11-04 08:59:29,538 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:59:39,423 DEBUG Dhanu => PING(20) 2019-11-04 08:59:49,548 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 08:59:59,441 DEBUG Dhanu => PING(20) 2019-11-04 09:00:09,556 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:00:19,457 DEBUG Dhanu => PING(20) 2019-11-04 09:00:29,566 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:00:39,478 DEBUG Dhanu => PING(20) 2019-11-04 09:00:49,574 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:00:59,497 DEBUG Dhanu => PING(20) 2019-11-04 09:01:09,583 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:01:19,519 DEBUG Dhanu => PING(20) 2019-11-04 09:01:29,590 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:01:39,537 DEBUG Dhanu => PING(20) 2019-11-04 09:01:49,598 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:01:59,557 DEBUG Dhanu => PING(20) 2019-11-04 09:02:09,606 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:02:19,577 DEBUG Dhanu => PING(20) 2019-11-04 09:02:29,612 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:02:39,590 DEBUG Dhanu => PING(20) 2019-11-04 09:02:49,620 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:02:59,611 DEBUG Dhanu => PING(20) 2019-11-04 09:03:09,630 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:03:19,633 DEBUG Dhanu => PING(20) 2019-11-04 09:03:29,638 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:03:39,650 DEBUG Dhanu => PING(20) 2019-11-04 09:03:49,646 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:03:59,665 DEBUG Dhanu => PING(20) 2019-11-04 09:04:09,654 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:04:19,687 DEBUG Dhanu => PING(20) 2019-11-04 09:04:29,660 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:04:39,706 DEBUG Dhanu => PING(20) 2019-11-04 09:04:49,668 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:04:59,722 DEBUG Dhanu => PING(20) 2019-11-04 09:05:09,676 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:05:19,740 DEBUG Dhanu => PING(20) 2019-11-04 09:05:29,686 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:05:39,763 DEBUG Dhanu => PING(20) 2019-11-04 09:05:49,694 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:05:59,786 DEBUG Dhanu => PING(20) 2019-11-04 09:06:09,703 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:06:19,810 DEBUG Dhanu => PING(20) 2019-11-04 09:06:29,712 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:06:39,833 DEBUG Dhanu => PING(20) 2019-11-04 09:06:49,720 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:06:59,853 DEBUG Dhanu => PING(20) 2019-11-04 09:07:09,730 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:07:19,877 DEBUG Dhanu => PING(20) 2019-11-04 09:07:29,738 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:07:39,901 DEBUG Dhanu => PING(20) 2019-11-04 09:07:49,746 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:07:59,919 DEBUG Dhanu => PING(20) 2019-11-04 09:08:09,754 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:08:19,943 DEBUG Dhanu => PING(20) 2019-11-04 09:08:29,761 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:08:39,963 DEBUG Dhanu => PING(20) 2019-11-04 09:08:49,768 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:08:59,988 DEBUG Dhanu => PING(20) 2019-11-04 09:09:09,776 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:09:20,011 DEBUG Dhanu => PING(20) 2019-11-04 09:09:29,784 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:09:40,035 DEBUG Dhanu => PING(20) 2019-11-04 09:09:49,792 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:10:00,057 DEBUG Dhanu => PING(20) 2019-11-04 09:10:09,800 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:10:20,079 DEBUG Dhanu => PING(20) 2019-11-04 09:10:29,808 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:10:40,101 DEBUG Dhanu => PING(20) 2019-11-04 09:10:49,817 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:11:00,121 DEBUG Dhanu => PING(20) 2019-11-04 09:11:09,827 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:11:20,143 DEBUG Dhanu => PING(20) 2019-11-04 09:11:29,836 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:11:40,163 DEBUG Dhanu => PING(20) 2019-11-04 09:11:49,846 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:12:00,187 DEBUG Dhanu => PING(20) 2019-11-04 09:12:09,854 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:12:20,205 DEBUG Dhanu => PING(20) 2019-11-04 09:12:29,862 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:12:40,225 DEBUG Dhanu => PING(20) 2019-11-04 09:12:49,872 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:13:00,247 DEBUG Dhanu => PING(20) 2019-11-04 09:13:09,881 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:13:20,269 DEBUG Dhanu => PING(20) 2019-11-04 09:13:29,890 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:13:40,293 DEBUG Dhanu => PING(20) 2019-11-04 09:13:49,900 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:14:00,313 DEBUG Dhanu => PING(20) 2019-11-04 09:14:09,908 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:14:20,338 DEBUG Dhanu => PING(20) 2019-11-04 09:14:29,918 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:14:40,359 DEBUG Dhanu => PING(20) 2019-11-04 09:14:49,927 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:15:00,382 DEBUG Dhanu => PING(20) 2019-11-04 09:15:09,936 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:15:20,399 DEBUG Dhanu => PING(20) 2019-11-04 09:15:29,945 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:15:40,419 DEBUG Dhanu => PING(20) 2019-11-04 09:15:49,954 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:16:00,444 DEBUG Dhanu => PING(20) 2019-11-04 09:16:09,963 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:16:20,465 DEBUG Dhanu => PING(20) 2019-11-04 09:16:29,972 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:16:40,487 DEBUG Dhanu => PING(20) 2019-11-04 09:16:49,981 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:17:00,510 DEBUG Dhanu => PING(20) 2019-11-04 09:17:09,991 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:17:20,529 DEBUG Dhanu => PING(20) 2019-11-04 09:17:29,999 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:17:40,553 DEBUG Dhanu => PING(20) 2019-11-04 09:17:50,007 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:18:00,573 DEBUG Dhanu => PING(20) 2019-11-04 09:18:10,015 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:18:20,595 DEBUG Dhanu => PING(20) 2019-11-04 09:18:30,023 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:18:40,617 DEBUG Dhanu => PING(20) 2019-11-04 09:18:50,031 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:19:00,637 DEBUG Dhanu => PING(20) 2019-11-04 09:19:10,038 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:19:20,657 DEBUG Dhanu => PING(20) 2019-11-04 09:19:30,045 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:19:40,682 DEBUG Dhanu => PING(20) 2019-11-04 09:19:50,054 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:20:00,706 DEBUG Dhanu => PING(20) 2019-11-04 09:20:10,061 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:20:20,730 DEBUG Dhanu => PING(20) 2019-11-04 09:20:30,069 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:20:40,752 DEBUG Dhanu => PING(20) 2019-11-04 09:20:50,076 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:21:00,774 DEBUG Dhanu => PING(20) 2019-11-04 09:21:10,084 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:21:20,795 DEBUG Dhanu => PING(20) 2019-11-04 09:21:30,092 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:21:40,814 DEBUG Dhanu => PING(20) 2019-11-04 09:21:50,099 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:22:00,838 DEBUG Dhanu => PING(20) 2019-11-04 09:22:10,108 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:22:20,859 DEBUG Dhanu => PING(20) 2019-11-04 09:22:30,116 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:22:40,879 DEBUG Dhanu => PING(20) 2019-11-04 09:22:50,123 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:23:00,903 DEBUG Dhanu => PING(20) 2019-11-04 09:23:10,131 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:23:20,927 DEBUG Dhanu => PING(20) 2019-11-04 09:23:30,141 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:23:40,947 DEBUG Dhanu => PING(20) 2019-11-04 09:23:50,149 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:24:00,969 DEBUG Dhanu => PING(20) 2019-11-04 09:24:10,156 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:24:20,990 DEBUG Dhanu => PING(20) 2019-11-04 09:24:30,164 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:24:41,010 DEBUG Dhanu => PING(20) 2019-11-04 09:24:50,172 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:25:01,029 DEBUG Dhanu => PING(20) 2019-11-04 09:25:10,180 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:25:21,053 DEBUG Dhanu => PING(20) 2019-11-04 09:25:30,188 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:25:41,078 DEBUG Dhanu => PING(20) 2019-11-04 09:25:50,196 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:26:01,101 DEBUG Dhanu => PING(20) 2019-11-04 09:26:10,204 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:26:21,125 DEBUG Dhanu => PING(20) 2019-11-04 09:26:30,212 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:26:41,143 DEBUG Dhanu => PING(20) 2019-11-04 09:26:50,219 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:27:01,161 DEBUG Dhanu => PING(20) 2019-11-04 09:27:10,226 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:27:21,186 DEBUG Dhanu => PING(20) 2019-11-04 09:27:30,232 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:27:41,206 DEBUG Dhanu => PING(20) 2019-11-04 09:27:50,240 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:28:01,227 DEBUG Dhanu => PING(20) 2019-11-04 09:28:10,247 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:28:21,251 DEBUG Dhanu => PING(20) 2019-11-04 09:28:30,255 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:33:21,538 DEBUG Dhanu => PING(20) 2019-11-04 09:33:30,362 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:33:41,560 DEBUG Dhanu => PING(20) 2019-11-04 09:33:50,370 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:34:01,579 DEBUG Dhanu => PING(20) 2019-11-04 09:34:10,379 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:34:21,599 DEBUG Dhanu => PING(20) 2019-11-04 09:34:30,387 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:34:41,623 DEBUG Dhanu => PING(20) 2019-11-04 09:34:50,395 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:35:01,641 DEBUG Dhanu => PING(20) 2019-11-04 09:35:10,402 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:35:21,662 DEBUG Dhanu => PING(20) 2019-11-04 09:35:30,407 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:35:41,681 DEBUG Dhanu => PING(20) 2019-11-04 09:35:50,412 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:36:01,697 DEBUG Dhanu => PING(20) 2019-11-04 09:36:10,420 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:36:21,719 DEBUG Dhanu => PING(20) 2019-11-04 09:36:30,426 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:36:41,743 DEBUG Dhanu => PING(20) 2019-11-04 09:36:50,434 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:37:01,761 DEBUG Dhanu => PING(20) 2019-11-04 09:37:10,442 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:37:21,782 DEBUG Dhanu => PING(20) 2019-11-04 09:37:30,451 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:37:41,801 DEBUG Dhanu => PING(20) 2019-11-04 09:37:50,457 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:37:53,610 INFO [POLL] Received a signal, leaving 2019-11-04 09:37:53,611 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 09:37:54,549 INFO [INIT] Dropping privileges 2019-11-04 09:37:54,549 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 09:37:54,549 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 09:37:54,551 INFO [INIT] Marking all workers as offline 2019-11-04 09:37:54,611 INFO [INIT] Starting encryption 2019-11-04 09:37:54,613 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 09:37:54,613 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 09:37:54,614 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 09:37:54,633 INFO [INIT] LAVA master has started. 2019-11-04 09:37:54,633 INFO [INIT] Using protocol version 3 2019-11-04 09:38:01,821 DEBUG Dhanu => PING(20) 2019-11-04 09:38:01,821 WARNING Unknown dispatcher (server crashed) 2019-11-04 09:38:14,645 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:38:21,846 DEBUG Dhanu => PING(20) 2019-11-04 09:38:34,655 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:38:41,870 DEBUG Dhanu => PING(20) 2019-11-04 09:38:54,663 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:39:01,889 DEBUG Dhanu => PING(20) 2019-11-04 09:39:14,671 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:39:21,910 DEBUG Dhanu => PING(20) 2019-11-04 09:39:34,681 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:39:41,925 DEBUG Dhanu => PING(20) 2019-11-04 09:39:54,692 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:40:01,970 DEBUG Dhanu => PING(20) 2019-11-04 09:40:14,704 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:40:21,998 DEBUG Dhanu => PING(20) 2019-11-04 09:40:34,716 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:40:42,017 DEBUG Dhanu => PING(20) 2019-11-04 09:40:54,728 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:41:02,033 DEBUG Dhanu => PING(20) 2019-11-04 09:41:14,740 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:41:22,051 DEBUG Dhanu => PING(20) 2019-11-04 09:41:34,745 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:41:42,066 DEBUG Dhanu => PING(20) 2019-11-04 09:41:54,754 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:42:02,091 DEBUG Dhanu => PING(20) 2019-11-04 09:42:14,761 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:42:22,115 DEBUG Dhanu => PING(20) 2019-11-04 09:42:34,768 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:42:42,133 DEBUG Dhanu => PING(20) 2019-11-04 09:42:54,779 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:43:02,158 DEBUG Dhanu => PING(20) 2019-11-04 09:43:14,791 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:43:22,182 DEBUG Dhanu => PING(20) 2019-11-04 09:43:34,803 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:43:42,202 DEBUG Dhanu => PING(20) 2019-11-04 09:43:54,812 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:44:02,217 DEBUG Dhanu => PING(20) 2019-11-04 09:44:14,824 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:44:22,235 DEBUG Dhanu => PING(20) 2019-11-04 09:44:34,836 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:44:42,253 DEBUG Dhanu => PING(20) 2019-11-04 09:44:54,849 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:45:02,269 DEBUG Dhanu => PING(20) 2019-11-04 09:45:14,861 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:45:22,565 DEBUG Dhanu => PING(20) 2019-11-04 09:45:34,924 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:45:42,586 DEBUG Dhanu => PING(20) 2019-11-04 09:45:54,931 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:46:02,605 DEBUG Dhanu => PING(20) 2019-11-04 09:46:14,939 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:46:22,629 DEBUG Dhanu => PING(20) 2019-11-04 09:46:34,947 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:46:42,645 DEBUG Dhanu => PING(20) 2019-11-04 09:46:54,949 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:47:02,665 DEBUG Dhanu => PING(20) 2019-11-04 09:47:14,957 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:47:22,685 DEBUG Dhanu => PING(20) 2019-11-04 09:47:34,964 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:47:42,700 DEBUG Dhanu => PING(20) 2019-11-04 09:47:54,971 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:48:02,721 DEBUG Dhanu => PING(20) 2019-11-04 09:48:14,977 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:48:22,742 DEBUG Dhanu => PING(20) 2019-11-04 09:48:34,984 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:48:42,763 DEBUG Dhanu => PING(20) 2019-11-04 09:48:54,991 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:49:02,781 DEBUG Dhanu => PING(20) 2019-11-04 09:49:14,998 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:49:22,799 DEBUG Dhanu => PING(20) 2019-11-04 09:49:35,000 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:49:42,821 DEBUG Dhanu => PING(20) 2019-11-04 09:49:55,008 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:50:02,840 DEBUG Dhanu => PING(20) 2019-11-04 09:50:15,015 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:50:22,857 DEBUG Dhanu => PING(20) 2019-11-04 09:50:35,025 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:50:42,873 DEBUG Dhanu => PING(20) 2019-11-04 09:50:55,032 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:51:02,889 DEBUG Dhanu => PING(20) 2019-11-04 09:51:15,044 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:51:22,911 DEBUG Dhanu => PING(20) 2019-11-04 09:51:35,052 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:51:42,931 DEBUG Dhanu => PING(20) 2019-11-04 09:51:55,064 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:52:02,953 DEBUG Dhanu => PING(20) 2019-11-04 09:52:15,076 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:52:22,975 DEBUG Dhanu => PING(20) 2019-11-04 09:52:35,084 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:52:42,997 DEBUG Dhanu => PING(20) 2019-11-04 09:52:55,096 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:53:03,013 DEBUG Dhanu => PING(20) 2019-11-04 09:53:15,108 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:53:23,031 DEBUG Dhanu => PING(20) 2019-11-04 09:53:35,120 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:53:43,051 DEBUG Dhanu => PING(20) 2019-11-04 09:53:55,133 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:54:03,076 DEBUG Dhanu => PING(20) 2019-11-04 09:54:15,145 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:54:23,100 DEBUG Dhanu => PING(20) 2019-11-04 09:54:35,157 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:54:43,125 DEBUG Dhanu => PING(20) 2019-11-04 09:54:55,169 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:55:03,146 DEBUG Dhanu => PING(20) 2019-11-04 09:55:15,180 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:55:23,171 DEBUG Dhanu => PING(20) 2019-11-04 09:55:35,188 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:55:43,189 DEBUG Dhanu => PING(20) 2019-11-04 09:55:55,196 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:56:03,207 DEBUG Dhanu => PING(20) 2019-11-04 09:56:15,208 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:56:23,231 DEBUG Dhanu => PING(20) 2019-11-04 09:56:35,215 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:56:43,256 DEBUG Dhanu => PING(20) 2019-11-04 09:56:55,222 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:57:03,280 DEBUG Dhanu => PING(20) 2019-11-04 09:57:15,232 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:57:23,305 DEBUG Dhanu => PING(20) 2019-11-04 09:57:35,243 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:57:43,329 DEBUG Dhanu => PING(20) 2019-11-04 09:57:55,255 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:58:03,353 DEBUG Dhanu => PING(20) 2019-11-04 09:58:15,264 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:58:23,373 DEBUG Dhanu => PING(20) 2019-11-04 09:58:35,271 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:58:43,412 DEBUG Dhanu => PING(20) 2019-11-04 09:58:55,282 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:59:03,436 DEBUG Dhanu => PING(20) 2019-11-04 09:59:15,294 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:59:23,461 DEBUG Dhanu => PING(20) 2019-11-04 09:59:35,306 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 09:59:43,483 DEBUG Dhanu => PING(20) 2019-11-04 09:59:55,318 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:00:03,506 DEBUG Dhanu => PING(20) 2019-11-04 10:00:15,329 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:00:23,525 DEBUG Dhanu => PING(20) 2019-11-04 10:00:35,336 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:00:43,547 DEBUG Dhanu => PING(20) 2019-11-04 10:00:55,347 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:01:03,569 DEBUG Dhanu => PING(20) 2019-11-04 10:01:15,358 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:01:23,593 DEBUG Dhanu => PING(20) 2019-11-04 10:01:35,406 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:01:43,811 DEBUG Dhanu => PING(20) 2019-11-04 10:01:55,417 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:02:03,832 DEBUG Dhanu => PING(20) 2019-11-04 10:02:15,428 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:02:23,855 DEBUG Dhanu => PING(20) 2019-11-04 10:02:35,440 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:02:43,877 DEBUG Dhanu => PING(20) 2019-11-04 10:02:55,452 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:03:03,899 DEBUG Dhanu => PING(20) 2019-11-04 10:03:15,463 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:03:23,919 DEBUG Dhanu => PING(20) 2019-11-04 10:03:35,474 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:03:43,939 DEBUG Dhanu => PING(20) 2019-11-04 10:03:55,485 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:04:03,964 DEBUG Dhanu => PING(20) 2019-11-04 10:04:15,496 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:04:23,988 DEBUG Dhanu => PING(20) 2019-11-04 10:04:35,507 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:04:44,011 DEBUG Dhanu => PING(20) 2019-11-04 10:04:55,515 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:05:04,034 DEBUG Dhanu => PING(20) 2019-11-04 10:05:15,522 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:05:24,049 DEBUG Dhanu => PING(20) 2019-11-04 10:05:35,530 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:05:44,066 DEBUG Dhanu => PING(20) 2019-11-04 10:05:55,540 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:06:04,082 DEBUG Dhanu => PING(20) 2019-11-04 10:06:15,548 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:06:24,101 DEBUG Dhanu => PING(20) 2019-11-04 10:06:35,556 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:06:44,117 DEBUG Dhanu => PING(20) 2019-11-04 10:06:55,567 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:07:04,142 DEBUG Dhanu => PING(20) 2019-11-04 10:07:15,576 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:07:24,158 DEBUG Dhanu => PING(20) 2019-11-04 10:07:35,587 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:07:44,181 DEBUG Dhanu => PING(20) 2019-11-04 10:07:55,598 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:08:04,199 DEBUG Dhanu => PING(20) 2019-11-04 10:08:15,608 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:08:24,218 DEBUG Dhanu => PING(20) 2019-11-04 10:08:35,616 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:08:44,233 DEBUG Dhanu => PING(20) 2019-11-04 10:08:55,624 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:09:04,249 DEBUG Dhanu => PING(20) 2019-11-04 10:09:15,632 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:09:24,265 DEBUG Dhanu => PING(20) 2019-11-04 10:09:35,640 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:09:44,285 DEBUG Dhanu => PING(20) 2019-11-04 10:09:55,651 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:10:04,301 DEBUG Dhanu => PING(20) 2019-11-04 10:10:15,660 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:10:24,322 DEBUG Dhanu => PING(20) 2019-11-04 10:10:35,668 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:10:44,337 DEBUG Dhanu => PING(20) 2019-11-04 10:10:55,677 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:11:04,355 DEBUG Dhanu => PING(20) 2019-11-04 10:11:15,684 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:11:24,374 DEBUG Dhanu => PING(20) 2019-11-04 10:11:35,692 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:11:44,393 DEBUG Dhanu => PING(20) 2019-11-04 10:11:55,703 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:12:04,411 DEBUG Dhanu => PING(20) 2019-11-04 10:12:15,713 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:12:24,429 DEBUG Dhanu => PING(20) 2019-11-04 10:12:35,724 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:12:44,445 DEBUG Dhanu => PING(20) 2019-11-04 10:12:55,732 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:13:04,461 DEBUG Dhanu => PING(20) 2019-11-04 10:13:15,743 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:13:24,477 DEBUG Dhanu => PING(20) 2019-11-04 10:13:35,754 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:13:44,493 DEBUG Dhanu => PING(20) 2019-11-04 10:13:55,764 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:14:04,510 DEBUG Dhanu => PING(20) 2019-11-04 10:14:15,775 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:14:24,534 DEBUG Dhanu => PING(20) 2019-11-04 10:14:35,783 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:14:44,558 DEBUG Dhanu => PING(20) 2019-11-04 10:14:55,791 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:15:04,576 DEBUG Dhanu => PING(20) 2019-11-04 10:15:15,799 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:15:24,601 DEBUG Dhanu => PING(20) 2019-11-04 10:15:35,807 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:15:44,622 DEBUG Dhanu => PING(20) 2019-11-04 10:15:55,817 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:16:04,642 DEBUG Dhanu => PING(20) 2019-11-04 10:16:15,828 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:16:24,668 DEBUG Dhanu => PING(20) 2019-11-04 10:16:35,836 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:16:44,685 DEBUG Dhanu => PING(20) 2019-11-04 10:16:55,847 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:17:04,706 DEBUG Dhanu => PING(20) 2019-11-04 10:17:15,855 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:17:24,721 DEBUG Dhanu => PING(20) 2019-11-04 10:17:35,864 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:17:44,737 DEBUG Dhanu => PING(20) 2019-11-04 10:17:55,872 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:18:04,749 DEBUG Dhanu => PING(20) 2019-11-04 10:18:15,880 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:18:24,771 DEBUG Dhanu => PING(20) 2019-11-04 10:18:35,888 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:18:44,789 DEBUG Dhanu => PING(20) 2019-11-04 10:18:55,894 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:19:04,803 DEBUG Dhanu => PING(20) 2019-11-04 10:19:15,901 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:19:24,821 DEBUG Dhanu => PING(20) 2019-11-04 10:19:35,907 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:19:44,838 DEBUG Dhanu => PING(20) 2019-11-04 10:19:55,912 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:20:04,857 DEBUG Dhanu => PING(20) 2019-11-04 10:20:15,920 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:20:24,873 DEBUG Dhanu => PING(20) 2019-11-04 10:20:35,928 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:20:44,893 DEBUG Dhanu => PING(20) 2019-11-04 10:20:55,936 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:21:04,909 DEBUG Dhanu => PING(20) 2019-11-04 10:21:15,942 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:21:24,925 DEBUG Dhanu => PING(20) 2019-11-04 10:21:35,949 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:21:44,941 DEBUG Dhanu => PING(20) 2019-11-04 10:21:55,956 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:22:04,957 DEBUG Dhanu => PING(20) 2019-11-04 10:22:15,964 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:22:24,977 DEBUG Dhanu => PING(20) 2019-11-04 10:22:35,971 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:22:44,993 DEBUG Dhanu => PING(20) 2019-11-04 10:22:55,980 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:23:05,009 DEBUG Dhanu => PING(20) 2019-11-04 10:23:15,989 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:23:25,025 DEBUG Dhanu => PING(20) 2019-11-04 10:23:35,996 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:23:45,043 DEBUG Dhanu => PING(20) 2019-11-04 10:23:56,006 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:24:05,061 DEBUG Dhanu => PING(20) 2019-11-04 10:24:16,012 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:24:25,078 DEBUG Dhanu => PING(20) 2019-11-04 10:24:36,022 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:24:45,092 DEBUG Dhanu => PING(20) 2019-11-04 10:24:56,033 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:25:05,116 DEBUG Dhanu => PING(20) 2019-11-04 10:25:16,040 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:25:25,135 DEBUG Dhanu => PING(20) 2019-11-04 10:25:36,048 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:25:45,150 DEBUG Dhanu => PING(20) 2019-11-04 10:25:56,056 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:26:05,171 DEBUG Dhanu => PING(20) 2019-11-04 10:26:16,066 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:26:25,195 DEBUG Dhanu => PING(20) 2019-11-04 10:26:36,077 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:26:45,217 DEBUG Dhanu => PING(20) 2019-11-04 10:26:56,087 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:27:05,241 DEBUG Dhanu => PING(20) 2019-11-04 10:27:16,098 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:27:25,266 DEBUG Dhanu => PING(20) 2019-11-04 10:27:36,106 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:27:45,290 DEBUG Dhanu => PING(20) 2019-11-04 10:27:56,114 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:28:05,315 DEBUG Dhanu => PING(20) 2019-11-04 10:28:16,119 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:28:25,339 DEBUG Dhanu => PING(20) 2019-11-04 10:28:36,128 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:28:45,364 DEBUG Dhanu => PING(20) 2019-11-04 10:28:56,139 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:29:05,389 DEBUG Dhanu => PING(20) 2019-11-04 10:29:16,149 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:29:25,410 DEBUG Dhanu => PING(20) 2019-11-04 10:29:36,159 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:29:45,433 DEBUG Dhanu => PING(20) 2019-11-04 10:29:56,169 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:30:05,453 DEBUG Dhanu => PING(20) 2019-11-04 10:30:16,179 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:30:25,475 DEBUG Dhanu => PING(20) 2019-11-04 10:30:36,188 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:30:45,499 DEBUG Dhanu => PING(20) 2019-11-04 10:30:56,199 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:31:05,522 DEBUG Dhanu => PING(20) 2019-11-04 10:31:16,209 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:31:25,546 DEBUG Dhanu => PING(20) 2019-11-04 10:31:36,216 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:31:45,565 DEBUG Dhanu => PING(20) 2019-11-04 10:31:56,227 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:32:05,581 DEBUG Dhanu => PING(20) 2019-11-04 10:32:16,238 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:32:25,599 DEBUG Dhanu => PING(20) 2019-11-04 10:32:36,246 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:32:45,619 DEBUG Dhanu => PING(20) 2019-11-04 10:32:56,254 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:33:05,637 DEBUG Dhanu => PING(20) 2019-11-04 10:33:16,262 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:33:25,655 DEBUG Dhanu => PING(20) 2019-11-04 10:33:36,270 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:33:45,677 DEBUG Dhanu => PING(20) 2019-11-04 10:33:56,280 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:34:05,694 DEBUG Dhanu => PING(20) 2019-11-04 10:34:16,288 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:34:25,714 DEBUG Dhanu => PING(20) 2019-11-04 10:34:36,296 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:34:45,729 DEBUG Dhanu => PING(20) 2019-11-04 10:34:56,304 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:35:05,745 DEBUG Dhanu => PING(20) 2019-11-04 10:35:16,308 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:35:25,765 DEBUG Dhanu => PING(20) 2019-11-04 10:35:36,315 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:35:45,785 DEBUG Dhanu => PING(20) 2019-11-04 10:35:56,323 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:36:05,801 DEBUG Dhanu => PING(20) 2019-11-04 10:36:16,332 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:36:25,821 DEBUG Dhanu => PING(20) 2019-11-04 10:36:36,340 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:36:45,838 DEBUG Dhanu => PING(20) 2019-11-04 10:36:56,348 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:37:05,856 DEBUG Dhanu => PING(20) 2019-11-04 10:37:16,357 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:37:25,874 DEBUG Dhanu => PING(20) 2019-11-04 10:37:36,364 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:37:45,893 DEBUG Dhanu => PING(20) 2019-11-04 10:37:56,373 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:38:05,909 DEBUG Dhanu => PING(20) 2019-11-04 10:38:16,380 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:38:25,925 DEBUG Dhanu => PING(20) 2019-11-04 10:38:36,391 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:38:45,944 DEBUG Dhanu => PING(20) 2019-11-04 10:38:56,400 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:39:05,961 DEBUG Dhanu => PING(20) 2019-11-04 10:39:16,408 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:39:25,977 DEBUG Dhanu => PING(20) 2019-11-04 10:39:36,418 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:39:45,993 DEBUG Dhanu => PING(20) 2019-11-04 10:39:56,428 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:40:06,009 DEBUG Dhanu => PING(20) 2019-11-04 10:40:16,436 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:40:26,025 DEBUG Dhanu => PING(20) 2019-11-04 10:40:36,444 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:40:46,041 DEBUG Dhanu => PING(20) 2019-11-04 10:40:56,454 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:41:06,057 DEBUG Dhanu => PING(20) 2019-11-04 10:41:16,460 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:41:26,074 DEBUG Dhanu => PING(20) 2019-11-04 10:41:36,468 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:41:46,093 DEBUG Dhanu => PING(20) 2019-11-04 10:41:56,476 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:42:06,109 DEBUG Dhanu => PING(20) 2019-11-04 10:42:16,486 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:42:26,125 DEBUG Dhanu => PING(20) 2019-11-04 10:42:36,492 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:42:46,143 DEBUG Dhanu => PING(20) 2019-11-04 10:42:56,500 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:43:06,161 DEBUG Dhanu => PING(20) 2019-11-04 10:43:16,508 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:43:26,177 DEBUG Dhanu => PING(20) 2019-11-04 10:43:36,516 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:43:46,193 DEBUG Dhanu => PING(20) 2019-11-04 10:43:56,524 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:44:06,207 DEBUG Dhanu => PING(20) 2019-11-04 10:44:16,534 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:44:26,225 DEBUG Dhanu => PING(20) 2019-11-04 10:44:36,540 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:44:46,244 DEBUG Dhanu => PING(20) 2019-11-04 10:44:56,548 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:45:06,261 DEBUG Dhanu => PING(20) 2019-11-04 10:45:16,556 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:45:26,278 DEBUG Dhanu => PING(20) 2019-11-04 10:45:36,564 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:45:46,298 DEBUG Dhanu => PING(20) 2019-11-04 10:45:56,572 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:46:06,317 DEBUG Dhanu => PING(20) 2019-11-04 10:46:16,580 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:46:26,333 DEBUG Dhanu => PING(20) 2019-11-04 10:46:36,588 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:46:46,350 DEBUG Dhanu => PING(20) 2019-11-04 10:46:56,596 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:47:06,369 DEBUG Dhanu => PING(20) 2019-11-04 10:47:16,606 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:47:26,385 DEBUG Dhanu => PING(20) 2019-11-04 10:47:36,616 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:47:46,401 DEBUG Dhanu => PING(20) 2019-11-04 10:47:56,624 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:48:06,417 DEBUG Dhanu => PING(20) 2019-11-04 10:48:16,635 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:48:26,433 DEBUG Dhanu => PING(20) 2019-11-04 10:48:36,641 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:48:46,453 DEBUG Dhanu => PING(20) 2019-11-04 10:48:56,648 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:49:06,475 DEBUG Dhanu => PING(20) 2019-11-04 10:49:16,657 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:49:26,493 DEBUG Dhanu => PING(20) 2019-11-04 10:49:36,664 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:49:46,509 DEBUG Dhanu => PING(20) 2019-11-04 10:49:56,672 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:50:06,525 DEBUG Dhanu => PING(20) 2019-11-04 10:50:16,680 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:50:26,542 DEBUG Dhanu => PING(20) 2019-11-04 10:50:36,685 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:50:46,557 DEBUG Dhanu => PING(20) 2019-11-04 10:50:56,691 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:51:06,573 DEBUG Dhanu => PING(20) 2019-11-04 10:51:16,700 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:51:26,590 DEBUG Dhanu => PING(20) 2019-11-04 10:51:36,708 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:51:46,608 DEBUG Dhanu => PING(20) 2019-11-04 10:51:56,716 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:52:06,629 DEBUG Dhanu => PING(20) 2019-11-04 10:52:16,724 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:52:26,645 DEBUG Dhanu => PING(20) 2019-11-04 10:52:36,732 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:52:46,661 DEBUG Dhanu => PING(20) 2019-11-04 10:52:56,740 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:53:06,681 DEBUG Dhanu => PING(20) 2019-11-04 10:53:16,748 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:53:26,697 DEBUG Dhanu => PING(20) 2019-11-04 10:53:36,756 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:53:46,713 DEBUG Dhanu => PING(20) 2019-11-04 10:53:56,766 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:54:06,729 DEBUG Dhanu => PING(20) 2019-11-04 10:54:55,164 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:54:55,168 DEBUG Dhanu => PING(20) 2019-11-04 10:55:15,180 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:55:15,193 DEBUG Dhanu => PING(20) 2019-11-04 10:55:35,196 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:55:35,209 DEBUG Dhanu => PING(20) 2019-11-04 10:55:55,212 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:55:55,225 DEBUG Dhanu => PING(20) 2019-11-04 10:56:15,232 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:56:15,241 DEBUG Dhanu => PING(20) 2019-11-04 10:56:35,251 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:56:35,257 DEBUG Dhanu => PING(20) 2019-11-04 10:56:45,369 INFO [POLL] Received a signal, leaving 2019-11-04 10:56:45,375 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 10:56:50,469 INFO [INIT] Dropping privileges 2019-11-04 10:56:50,470 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 10:56:50,470 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 10:56:50,530 INFO [INIT] Marking all workers as offline 2019-11-04 10:56:50,609 INFO [INIT] Starting encryption 2019-11-04 10:56:50,610 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 10:56:50,674 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 10:56:50,675 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 10:56:50,687 INFO [INIT] LAVA master has started. 2019-11-04 10:56:50,687 INFO [INIT] Using protocol version 3 2019-11-04 10:56:55,273 DEBUG Dhanu => PING(20) 2019-11-04 10:56:55,273 WARNING Unknown dispatcher (server crashed) 2019-11-04 10:57:10,700 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:57:15,162 INFO Dhanu => HELLO 2019-11-04 10:57:15,162 WARNING Dispatcher has RESTARTED 2019-11-04 10:57:20,165 DEBUG Dhanu => PING(20) 2019-11-04 10:57:30,710 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:57:40,237 DEBUG Dhanu => PING(20) 2019-11-04 10:57:50,720 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 10:57:56,822 INFO [POLL] Received a signal, leaving 2019-11-04 10:57:56,826 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:31:39,633 INFO [INIT] Dropping privileges 2019-11-04 16:31:39,647 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:31:39,647 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:31:39,722 INFO [INIT] Marking all workers as offline 2019-11-04 16:31:39,887 INFO [INIT] Starting encryption 2019-11-04 16:31:39,888 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:31:39,909 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:31:39,909 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:31:39,917 INFO [INIT] LAVA master has started. 2019-11-04 16:31:39,918 INFO [INIT] Using protocol version 3 2019-11-04 16:31:40,051 INFO Dhanu => HELLO 2019-11-04 16:31:40,052 WARNING New dispatcher 2019-11-04 16:31:40,120 INFO Dhanu => HELLO_RETRY 2019-11-04 16:31:40,121 WARNING Dispatcher was not confirmed 2019-11-04 16:31:40,154 INFO Dhanu => HELLO_RETRY 2019-11-04 16:31:40,154 WARNING Dispatcher was not confirmed 2019-11-04 16:31:40,209 INFO Dhanu => HELLO_RETRY 2019-11-04 16:31:40,209 WARNING Dispatcher was not confirmed 2019-11-04 16:31:40,242 INFO Dhanu => HELLO_RETRY 2019-11-04 16:31:40,243 WARNING Dispatcher was not confirmed 2019-11-04 16:31:45,250 DEBUG Dhanu => PING(20) 2019-11-04 16:31:59,933 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:32:05,268 DEBUG Dhanu => PING(20) 2019-11-04 16:32:20,153 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:32:25,292 DEBUG Dhanu => PING(20) 2019-11-04 16:32:40,161 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:32:45,316 DEBUG Dhanu => PING(20) 2019-11-04 16:33:00,168 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:33:05,338 DEBUG Dhanu => PING(20) 2019-11-04 16:33:20,175 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:33:25,362 DEBUG Dhanu => PING(20) 2019-11-04 16:33:40,183 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:33:45,382 DEBUG Dhanu => PING(20) 2019-11-04 16:34:00,189 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:34:05,399 DEBUG Dhanu => PING(20) 2019-11-04 16:34:20,192 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:34:25,415 DEBUG Dhanu => PING(20) 2019-11-04 16:34:40,196 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:34:45,430 DEBUG Dhanu => PING(20) 2019-11-04 16:35:00,200 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:35:05,450 DEBUG Dhanu => PING(20) 2019-11-04 16:35:20,204 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:35:25,468 DEBUG Dhanu => PING(20) 2019-11-04 16:35:40,211 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:35:45,508 DEBUG Dhanu => PING(20) 2019-11-04 16:36:00,218 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:36:05,531 DEBUG Dhanu => PING(20) 2019-11-04 16:36:20,226 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:36:25,552 DEBUG Dhanu => PING(20) 2019-11-04 16:36:40,235 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:36:45,571 DEBUG Dhanu => PING(20) 2019-11-04 16:37:00,243 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:37:05,593 DEBUG Dhanu => PING(20) 2019-11-04 16:37:20,250 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:37:25,616 DEBUG Dhanu => PING(20) 2019-11-04 16:37:40,257 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:37:45,640 DEBUG Dhanu => PING(20) 2019-11-04 16:38:00,265 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:38:01,441 DEBUG [AUTH] Reloading certificates from /etc/lava-dispatcher/certificates.d 2019-11-04 16:38:01,441 DEBUG [AUTH] Reloading certificates from /etc/lava-dispatcher/certificates.d 2019-11-04 16:38:01,442 DEBUG [AUTH] Reloading certificates from /etc/lava-dispatcher/certificates.d 2019-11-04 16:38:05,665 DEBUG Dhanu => PING(20) 2019-11-04 16:38:20,273 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:38:25,687 DEBUG Dhanu => PING(20) 2019-11-04 16:38:40,281 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:38:45,707 DEBUG Dhanu => PING(20) 2019-11-04 16:39:00,288 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:39:05,728 DEBUG Dhanu => PING(20) 2019-11-04 16:39:20,298 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:39:25,751 DEBUG Dhanu => PING(20) 2019-11-04 16:39:40,307 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:39:45,777 DEBUG Dhanu => PING(20) 2019-11-04 16:40:00,315 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:40:05,799 DEBUG Dhanu => PING(20) 2019-11-04 16:40:20,323 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:40:25,822 DEBUG Dhanu => PING(20) 2019-11-04 16:40:36,259 DEBUG lava-logs => PING(20) 2019-11-04 16:40:36,285 DEBUG lava-logs => PING(20) 2019-11-04 16:40:40,331 INFO scheduling health checks: 2019-11-04 16:40:40,528 INFO scheduling jobs: 2019-11-04 16:40:40,531 DEBUG - kvm 2019-11-04 16:40:40,743 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:40:40,743 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:40:40,807 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:40:42,688 INFO [INIT] Dropping privileges 2019-11-04 16:40:42,689 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:40:42,689 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:40:42,691 INFO [INIT] Marking all workers as offline 2019-11-04 16:40:42,719 INFO [INIT] Starting encryption 2019-11-04 16:40:42,721 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:40:42,722 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:40:42,722 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:40:42,736 INFO [INIT] LAVA master has started. 2019-11-04 16:40:42,737 INFO [INIT] Using protocol version 3 2019-11-04 16:40:45,842 DEBUG Dhanu => PING(20) 2019-11-04 16:40:45,842 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:40:56,302 DEBUG lava-logs => PING(20) 2019-11-04 16:41:02,743 INFO scheduling health checks: 2019-11-04 16:41:02,787 INFO scheduling jobs: 2019-11-04 16:41:02,789 DEBUG - kvm 2019-11-04 16:41:02,864 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:41:02,864 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:41:02,869 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:41:04,107 INFO [INIT] Dropping privileges 2019-11-04 16:41:04,107 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:41:04,108 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:41:04,109 INFO [INIT] Marking all workers as offline 2019-11-04 16:41:04,143 INFO [INIT] Starting encryption 2019-11-04 16:41:04,144 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:41:04,144 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:41:04,145 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:41:04,158 INFO [INIT] LAVA master has started. 2019-11-04 16:41:04,158 INFO [INIT] Using protocol version 3 2019-11-04 16:41:05,862 DEBUG Dhanu => PING(20) 2019-11-04 16:41:05,863 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:41:16,315 DEBUG lava-logs => PING(20) 2019-11-04 16:41:18,967 INFO Dhanu => HELLO 2019-11-04 16:41:18,968 WARNING Dispatcher has RESTARTED 2019-11-04 16:41:23,974 DEBUG Dhanu => PING(20) 2019-11-04 16:41:24,160 INFO scheduling health checks: 2019-11-04 16:41:24,200 INFO scheduling jobs: 2019-11-04 16:41:24,202 DEBUG - kvm 2019-11-04 16:41:24,283 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:41:24,284 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:41:24,288 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:41:25,526 INFO [INIT] Dropping privileges 2019-11-04 16:41:25,527 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:41:25,527 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:41:25,528 INFO [INIT] Marking all workers as offline 2019-11-04 16:41:25,556 INFO [INIT] Starting encryption 2019-11-04 16:41:25,557 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:41:25,558 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:41:25,558 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:41:25,573 INFO [INIT] LAVA master has started. 2019-11-04 16:41:25,574 INFO [INIT] Using protocol version 3 2019-11-04 16:41:36,327 DEBUG lava-logs => PING(20) 2019-11-04 16:41:43,998 DEBUG Dhanu => PING(20) 2019-11-04 16:41:43,998 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:41:45,575 INFO scheduling health checks: 2019-11-04 16:41:45,613 INFO scheduling jobs: 2019-11-04 16:41:45,615 DEBUG - kvm 2019-11-04 16:41:45,696 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:41:45,696 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:41:45,701 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:41:46,980 INFO [INIT] Dropping privileges 2019-11-04 16:41:46,981 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:41:46,981 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:41:46,983 INFO [INIT] Marking all workers as offline 2019-11-04 16:41:47,014 INFO [INIT] Starting encryption 2019-11-04 16:41:47,015 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:41:47,015 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:41:47,015 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:41:47,025 INFO [INIT] LAVA master has started. 2019-11-04 16:41:47,026 INFO [INIT] Using protocol version 3 2019-11-04 16:41:56,346 DEBUG lava-logs => PING(20) 2019-11-04 16:42:04,014 DEBUG Dhanu => PING(20) 2019-11-04 16:42:04,014 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:42:07,029 INFO scheduling health checks: 2019-11-04 16:42:07,071 INFO scheduling jobs: 2019-11-04 16:42:07,073 DEBUG - kvm 2019-11-04 16:42:07,151 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:42:07,151 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:42:07,156 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:42:08,538 INFO [INIT] Dropping privileges 2019-11-04 16:42:08,538 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:42:08,538 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:42:08,540 INFO [INIT] Marking all workers as offline 2019-11-04 16:42:08,572 INFO [INIT] Starting encryption 2019-11-04 16:42:08,574 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:42:08,575 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:42:08,575 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:42:08,587 INFO [INIT] LAVA master has started. 2019-11-04 16:42:08,588 INFO [INIT] Using protocol version 3 2019-11-04 16:42:16,359 DEBUG lava-logs => PING(20) 2019-11-04 16:42:24,026 DEBUG Dhanu => PING(20) 2019-11-04 16:42:24,026 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:42:28,590 INFO scheduling health checks: 2019-11-04 16:42:28,684 INFO scheduling jobs: 2019-11-04 16:42:28,687 DEBUG - kvm 2019-11-04 16:42:28,772 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:42:28,772 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:42:28,778 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:42:30,239 INFO [INIT] Dropping privileges 2019-11-04 16:42:30,240 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:42:30,240 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:42:30,242 INFO [INIT] Marking all workers as offline 2019-11-04 16:42:30,263 INFO [INIT] Starting encryption 2019-11-04 16:42:30,263 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:42:30,263 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:42:30,263 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:42:30,272 INFO [INIT] LAVA master has started. 2019-11-04 16:42:30,272 INFO [INIT] Using protocol version 3 2019-11-04 16:42:36,377 DEBUG lava-logs => PING(20) 2019-11-04 16:42:44,042 DEBUG Dhanu => PING(20) 2019-11-04 16:42:44,043 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:42:50,278 INFO scheduling health checks: 2019-11-04 16:42:50,343 INFO scheduling jobs: 2019-11-04 16:42:50,345 DEBUG - kvm 2019-11-04 16:42:50,430 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:42:50,430 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:42:50,439 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:42:51,791 INFO [INIT] Dropping privileges 2019-11-04 16:42:51,791 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:42:51,791 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:42:51,793 INFO [INIT] Marking all workers as offline 2019-11-04 16:42:51,809 INFO [INIT] Starting encryption 2019-11-04 16:42:51,811 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:42:51,811 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:42:51,811 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:42:51,823 INFO [INIT] LAVA master has started. 2019-11-04 16:42:51,823 INFO [INIT] Using protocol version 3 2019-11-04 16:42:56,392 DEBUG lava-logs => PING(20) 2019-11-04 16:43:04,057 DEBUG Dhanu => PING(20) 2019-11-04 16:43:04,057 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:43:11,830 INFO scheduling health checks: 2019-11-04 16:43:11,889 INFO scheduling jobs: 2019-11-04 16:43:11,890 DEBUG - kvm 2019-11-04 16:43:11,991 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:43:11,991 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:43:11,996 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:43:13,471 INFO [INIT] Dropping privileges 2019-11-04 16:43:13,472 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:43:13,472 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:43:13,474 INFO [INIT] Marking all workers as offline 2019-11-04 16:43:13,501 INFO [INIT] Starting encryption 2019-11-04 16:43:13,504 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:43:13,504 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:43:13,505 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:43:13,517 INFO [INIT] LAVA master has started. 2019-11-04 16:43:13,517 INFO [INIT] Using protocol version 3 2019-11-04 16:43:16,405 DEBUG lava-logs => PING(20) 2019-11-04 16:43:24,072 DEBUG Dhanu => PING(20) 2019-11-04 16:43:24,073 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:43:33,526 INFO scheduling health checks: 2019-11-04 16:43:33,580 INFO scheduling jobs: 2019-11-04 16:43:33,581 DEBUG - kvm 2019-11-04 16:43:33,659 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:43:33,659 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:43:33,666 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:43:34,876 INFO [INIT] Dropping privileges 2019-11-04 16:43:34,877 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:43:34,877 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:43:34,878 INFO [INIT] Marking all workers as offline 2019-11-04 16:43:34,914 INFO [INIT] Starting encryption 2019-11-04 16:43:34,917 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:43:34,918 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:43:34,918 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:43:34,932 INFO [INIT] LAVA master has started. 2019-11-04 16:43:34,932 INFO [INIT] Using protocol version 3 2019-11-04 16:43:36,417 DEBUG lava-logs => PING(20) 2019-11-04 16:43:44,096 DEBUG Dhanu => PING(20) 2019-11-04 16:43:44,097 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:43:54,943 INFO scheduling health checks: 2019-11-04 16:43:55,004 INFO scheduling jobs: 2019-11-04 16:43:55,006 DEBUG - kvm 2019-11-04 16:43:55,081 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:43:55,081 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:43:55,091 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:43:56,381 INFO [INIT] Dropping privileges 2019-11-04 16:43:56,381 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:43:56,381 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:43:56,383 INFO [INIT] Marking all workers as offline 2019-11-04 16:43:56,438 INFO [INIT] Starting encryption 2019-11-04 16:43:56,440 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:43:56,440 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:43:56,441 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:43:56,455 INFO [INIT] LAVA master has started. 2019-11-04 16:43:56,455 INFO [INIT] Using protocol version 3 2019-11-04 16:43:56,472 DEBUG lava-logs => PING(20) 2019-11-04 16:44:04,119 DEBUG Dhanu => PING(20) 2019-11-04 16:44:04,119 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:44:16,468 INFO scheduling health checks: 2019-11-04 16:44:16,518 INFO scheduling jobs: 2019-11-04 16:44:16,520 DEBUG - kvm 2019-11-04 16:44:16,595 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:44:16,595 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:44:16,600 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:44:17,855 INFO [INIT] Dropping privileges 2019-11-04 16:44:17,855 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:44:17,856 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:44:17,858 INFO [INIT] Marking all workers as offline 2019-11-04 16:44:17,885 INFO [INIT] Starting encryption 2019-11-04 16:44:17,886 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:44:17,887 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:44:17,887 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:44:17,901 INFO [INIT] LAVA master has started. 2019-11-04 16:44:17,901 INFO [INIT] Using protocol version 3 2019-11-04 16:44:24,138 DEBUG Dhanu => PING(20) 2019-11-04 16:44:24,139 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:44:36,506 DEBUG lava-logs => PING(20) 2019-11-04 16:44:37,902 INFO scheduling health checks: 2019-11-04 16:44:37,931 INFO scheduling jobs: 2019-11-04 16:44:37,934 DEBUG - kvm 2019-11-04 16:44:38,018 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:44:38,019 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:44:38,023 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:44:39,508 INFO [INIT] Dropping privileges 2019-11-04 16:44:39,508 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:44:39,509 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:44:39,510 INFO [INIT] Marking all workers as offline 2019-11-04 16:44:39,531 INFO [INIT] Starting encryption 2019-11-04 16:44:39,534 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:44:39,534 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:44:39,534 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:44:39,547 INFO [INIT] LAVA master has started. 2019-11-04 16:44:39,547 INFO [INIT] Using protocol version 3 2019-11-04 16:44:44,155 DEBUG Dhanu => PING(20) 2019-11-04 16:44:44,155 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:44:56,522 DEBUG lava-logs => PING(20) 2019-11-04 16:44:59,550 INFO scheduling health checks: 2019-11-04 16:44:59,599 INFO scheduling jobs: 2019-11-04 16:44:59,601 DEBUG - kvm 2019-11-04 16:44:59,694 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:44:59,694 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:44:59,703 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:45:00,915 INFO [INIT] Dropping privileges 2019-11-04 16:45:00,915 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:45:00,916 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:45:00,917 INFO [INIT] Marking all workers as offline 2019-11-04 16:45:01,068 INFO [INIT] Starting encryption 2019-11-04 16:45:01,069 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:45:01,070 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:45:01,070 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:45:01,082 INFO [INIT] LAVA master has started. 2019-11-04 16:45:01,082 INFO [INIT] Using protocol version 3 2019-11-04 16:45:04,171 DEBUG Dhanu => PING(20) 2019-11-04 16:45:04,174 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:45:16,537 DEBUG lava-logs => PING(20) 2019-11-04 16:45:21,087 INFO scheduling health checks: 2019-11-04 16:45:21,168 INFO scheduling jobs: 2019-11-04 16:45:21,170 DEBUG - kvm 2019-11-04 16:45:21,245 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:45:21,246 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:45:21,250 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:45:22,353 INFO [INIT] Dropping privileges 2019-11-04 16:45:22,353 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:45:22,353 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:45:22,355 INFO [INIT] Marking all workers as offline 2019-11-04 16:45:22,391 INFO [INIT] Starting encryption 2019-11-04 16:45:22,393 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:45:22,394 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:45:22,394 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:45:22,408 INFO [INIT] LAVA master has started. 2019-11-04 16:45:22,408 INFO [INIT] Using protocol version 3 2019-11-04 16:45:24,192 DEBUG Dhanu => PING(20) 2019-11-04 16:45:24,193 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:45:36,550 DEBUG lava-logs => PING(20) 2019-11-04 16:45:42,414 INFO scheduling health checks: 2019-11-04 16:45:42,493 INFO scheduling jobs: 2019-11-04 16:45:42,494 DEBUG - kvm 2019-11-04 16:45:42,568 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:45:42,568 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:45:42,573 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:45:43,859 INFO [INIT] Dropping privileges 2019-11-04 16:45:43,859 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:45:43,859 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:45:43,861 INFO [INIT] Marking all workers as offline 2019-11-04 16:45:43,894 INFO [INIT] Starting encryption 2019-11-04 16:45:43,896 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:45:43,896 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:45:43,896 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:45:43,912 INFO [INIT] LAVA master has started. 2019-11-04 16:45:43,912 INFO [INIT] Using protocol version 3 2019-11-04 16:45:44,206 DEBUG Dhanu => PING(20) 2019-11-04 16:45:44,207 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:45:56,563 DEBUG lava-logs => PING(20) 2019-11-04 16:46:03,920 INFO scheduling health checks: 2019-11-04 16:46:03,973 INFO scheduling jobs: 2019-11-04 16:46:03,975 DEBUG - kvm 2019-11-04 16:46:04,050 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:46:04,051 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:46:04,055 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:46:05,361 INFO [INIT] Dropping privileges 2019-11-04 16:46:05,361 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:46:05,362 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:46:05,363 INFO [INIT] Marking all workers as offline 2019-11-04 16:46:05,395 INFO [INIT] Starting encryption 2019-11-04 16:46:05,397 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:46:05,397 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:46:05,397 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:46:05,410 INFO [INIT] LAVA master has started. 2019-11-04 16:46:05,410 INFO [INIT] Using protocol version 3 2019-11-04 16:46:05,503 DEBUG Dhanu => PING(20) 2019-11-04 16:46:05,504 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:46:16,576 DEBUG lava-logs => PING(20) 2019-11-04 16:46:25,419 INFO scheduling health checks: 2019-11-04 16:46:25,475 INFO scheduling jobs: 2019-11-04 16:46:25,476 DEBUG - kvm 2019-11-04 16:46:25,550 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:46:25,550 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:46:25,554 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:46:26,843 INFO [INIT] Dropping privileges 2019-11-04 16:46:26,843 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:46:26,843 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:46:26,845 INFO [INIT] Marking all workers as offline 2019-11-04 16:46:26,875 INFO [INIT] Starting encryption 2019-11-04 16:46:26,876 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:46:26,877 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:46:26,877 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:46:26,892 INFO [INIT] LAVA master has started. 2019-11-04 16:46:26,892 INFO [INIT] Using protocol version 3 2019-11-04 16:46:36,596 DEBUG lava-logs => PING(20) 2019-11-04 16:46:45,540 DEBUG Dhanu => PING(20) 2019-11-04 16:46:45,541 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:46:46,894 INFO scheduling health checks: 2019-11-04 16:46:46,966 INFO scheduling jobs: 2019-11-04 16:46:46,968 DEBUG - kvm 2019-11-04 16:46:47,042 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:46:47,042 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:46:47,046 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:46:48,408 INFO [INIT] Dropping privileges 2019-11-04 16:46:48,408 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:46:48,408 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:46:48,410 INFO [INIT] Marking all workers as offline 2019-11-04 16:46:48,455 INFO [INIT] Starting encryption 2019-11-04 16:46:48,455 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:46:48,455 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:46:48,456 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:46:48,464 INFO [INIT] LAVA master has started. 2019-11-04 16:46:48,464 INFO [INIT] Using protocol version 3 2019-11-04 16:46:56,617 DEBUG lava-logs => PING(20) 2019-11-04 16:47:05,562 DEBUG Dhanu => PING(20) 2019-11-04 16:47:05,563 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:47:08,466 INFO scheduling health checks: 2019-11-04 16:47:08,534 INFO scheduling jobs: 2019-11-04 16:47:08,536 DEBUG - kvm 2019-11-04 16:47:08,610 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:47:08,610 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:47:08,614 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:47:09,872 INFO [INIT] Dropping privileges 2019-11-04 16:47:09,872 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:47:09,872 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:47:09,874 INFO [INIT] Marking all workers as offline 2019-11-04 16:47:09,902 INFO [INIT] Starting encryption 2019-11-04 16:47:09,905 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:47:09,905 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:47:09,906 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:47:09,922 INFO [INIT] LAVA master has started. 2019-11-04 16:47:09,922 INFO [INIT] Using protocol version 3 2019-11-04 16:47:16,637 DEBUG lava-logs => PING(20) 2019-11-04 16:47:25,584 DEBUG Dhanu => PING(20) 2019-11-04 16:47:25,585 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:47:29,926 INFO scheduling health checks: 2019-11-04 16:47:29,992 INFO scheduling jobs: 2019-11-04 16:47:29,994 DEBUG - kvm 2019-11-04 16:47:30,070 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:47:30,070 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:47:30,075 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:47:31,340 INFO [INIT] Dropping privileges 2019-11-04 16:47:31,341 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:47:31,341 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:47:31,342 INFO [INIT] Marking all workers as offline 2019-11-04 16:47:31,371 INFO [INIT] Starting encryption 2019-11-04 16:47:31,372 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:47:31,372 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:47:31,373 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:47:31,385 INFO [INIT] LAVA master has started. 2019-11-04 16:47:31,385 INFO [INIT] Using protocol version 3 2019-11-04 16:47:36,653 DEBUG lava-logs => PING(20) 2019-11-04 16:47:45,607 DEBUG Dhanu => PING(20) 2019-11-04 16:47:45,607 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:47:51,391 INFO scheduling health checks: 2019-11-04 16:47:51,472 INFO scheduling jobs: 2019-11-04 16:47:51,474 DEBUG - kvm 2019-11-04 16:47:51,553 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:47:51,553 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:47:51,563 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:47:52,878 INFO [INIT] Dropping privileges 2019-11-04 16:47:52,878 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:47:52,878 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:47:52,880 INFO [INIT] Marking all workers as offline 2019-11-04 16:47:52,940 INFO [INIT] Starting encryption 2019-11-04 16:47:52,942 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:47:52,942 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:47:52,943 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:47:52,957 INFO [INIT] LAVA master has started. 2019-11-04 16:47:52,957 INFO [INIT] Using protocol version 3 2019-11-04 16:47:56,670 DEBUG lava-logs => PING(20) 2019-11-04 16:48:05,628 DEBUG Dhanu => PING(20) 2019-11-04 16:48:05,629 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:48:12,958 INFO scheduling health checks: 2019-11-04 16:48:13,020 INFO scheduling jobs: 2019-11-04 16:48:13,022 DEBUG - kvm 2019-11-04 16:48:13,118 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:48:13,118 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:48:13,125 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:48:14,360 INFO [INIT] Dropping privileges 2019-11-04 16:48:14,360 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:48:14,361 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:48:14,362 INFO [INIT] Marking all workers as offline 2019-11-04 16:48:14,397 INFO [INIT] Starting encryption 2019-11-04 16:48:14,399 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:48:14,399 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:48:14,400 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:48:14,414 INFO [INIT] LAVA master has started. 2019-11-04 16:48:14,415 INFO [INIT] Using protocol version 3 2019-11-04 16:48:16,685 DEBUG lava-logs => PING(20) 2019-11-04 16:48:25,649 DEBUG Dhanu => PING(20) 2019-11-04 16:48:25,650 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:48:34,418 INFO scheduling health checks: 2019-11-04 16:48:34,465 INFO scheduling jobs: 2019-11-04 16:48:34,467 DEBUG - kvm 2019-11-04 16:48:34,541 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:48:34,541 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:48:34,545 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:48:35,852 INFO [INIT] Dropping privileges 2019-11-04 16:48:35,853 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:48:35,853 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:48:35,854 INFO [INIT] Marking all workers as offline 2019-11-04 16:48:35,889 INFO [INIT] Starting encryption 2019-11-04 16:48:35,891 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:48:35,891 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:48:35,892 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:48:35,906 INFO [INIT] LAVA master has started. 2019-11-04 16:48:35,907 INFO [INIT] Using protocol version 3 2019-11-04 16:48:36,699 DEBUG lava-logs => PING(20) 2019-11-04 16:48:45,669 DEBUG Dhanu => PING(20) 2019-11-04 16:48:45,670 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:48:55,917 INFO scheduling health checks: 2019-11-04 16:48:55,978 INFO scheduling jobs: 2019-11-04 16:48:55,981 DEBUG - kvm 2019-11-04 16:48:56,058 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:48:56,058 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:48:56,067 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:48:57,368 INFO [INIT] Dropping privileges 2019-11-04 16:48:57,368 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:48:57,369 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:48:57,370 INFO [INIT] Marking all workers as offline 2019-11-04 16:48:57,424 INFO [INIT] Starting encryption 2019-11-04 16:48:57,426 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:48:57,426 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:48:57,427 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:48:57,440 INFO [INIT] LAVA master has started. 2019-11-04 16:48:57,441 INFO [INIT] Using protocol version 3 2019-11-04 16:48:57,529 DEBUG lava-logs => PING(20) 2019-11-04 16:49:05,687 DEBUG Dhanu => PING(20) 2019-11-04 16:49:05,688 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:49:17,452 INFO scheduling health checks: 2019-11-04 16:49:17,514 INFO scheduling jobs: 2019-11-04 16:49:17,515 DEBUG - kvm 2019-11-04 16:49:17,591 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:49:17,591 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:49:17,601 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:49:18,851 INFO [INIT] Dropping privileges 2019-11-04 16:49:18,851 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:49:18,851 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:49:18,853 INFO [INIT] Marking all workers as offline 2019-11-04 16:49:18,882 INFO [INIT] Starting encryption 2019-11-04 16:49:18,884 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:49:18,885 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:49:18,885 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:49:18,902 INFO [INIT] LAVA master has started. 2019-11-04 16:49:18,902 INFO [INIT] Using protocol version 3 2019-11-04 16:49:25,707 DEBUG Dhanu => PING(20) 2019-11-04 16:49:25,707 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:49:38,912 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 16:49:39,750 DEBUG lava-logs => PING(20) 2019-11-04 16:49:39,772 DEBUG lava-logs => PING(20) 2019-11-04 16:49:39,896 INFO [POLL] Received a signal, leaving 2019-11-04 16:49:39,896 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:49:40,853 INFO [INIT] Dropping privileges 2019-11-04 16:49:40,854 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:49:40,854 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:49:40,856 INFO [INIT] Marking all workers as offline 2019-11-04 16:49:40,895 INFO [INIT] Starting encryption 2019-11-04 16:49:40,897 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:49:40,897 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:49:40,898 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:49:40,912 INFO [INIT] LAVA master has started. 2019-11-04 16:49:40,913 INFO [INIT] Using protocol version 3 2019-11-04 16:49:45,720 DEBUG Dhanu => PING(20) 2019-11-04 16:49:45,721 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:49:49,049 INFO Dhanu => HELLO 2019-11-04 16:49:49,049 WARNING Dispatcher has RESTARTED 2019-11-04 16:49:54,055 DEBUG Dhanu => PING(20) 2019-11-04 16:49:59,790 DEBUG lava-logs => PING(20) 2019-11-04 16:50:00,914 INFO scheduling health checks: 2019-11-04 16:50:00,964 INFO scheduling jobs: 2019-11-04 16:50:00,966 DEBUG - kvm 2019-11-04 16:50:01,046 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:50:01,046 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:50:01,051 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:50:02,421 INFO [INIT] Dropping privileges 2019-11-04 16:50:02,421 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:50:02,421 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:50:02,423 INFO [INIT] Marking all workers as offline 2019-11-04 16:50:02,464 INFO [INIT] Starting encryption 2019-11-04 16:50:02,465 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:50:02,466 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:50:02,466 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:50:02,480 INFO [INIT] LAVA master has started. 2019-11-04 16:50:02,480 INFO [INIT] Using protocol version 3 2019-11-04 16:50:14,068 DEBUG Dhanu => PING(20) 2019-11-04 16:50:14,069 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:50:19,809 DEBUG lava-logs => PING(20) 2019-11-04 16:50:22,483 INFO scheduling health checks: 2019-11-04 16:50:22,565 INFO scheduling jobs: 2019-11-04 16:50:22,567 DEBUG - kvm 2019-11-04 16:50:22,652 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:50:22,653 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:50:22,657 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:50:23,865 INFO [INIT] Dropping privileges 2019-11-04 16:50:23,866 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:50:23,866 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:50:23,868 INFO [INIT] Marking all workers as offline 2019-11-04 16:50:23,911 INFO [INIT] Starting encryption 2019-11-04 16:50:23,913 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:50:23,914 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:50:23,914 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:50:23,929 INFO [INIT] LAVA master has started. 2019-11-04 16:50:23,929 INFO [INIT] Using protocol version 3 2019-11-04 16:50:34,086 DEBUG Dhanu => PING(20) 2019-11-04 16:50:34,086 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:50:39,825 DEBUG lava-logs => PING(20) 2019-11-04 16:50:43,930 INFO scheduling health checks: 2019-11-04 16:50:43,978 INFO scheduling jobs: 2019-11-04 16:50:43,980 DEBUG - kvm 2019-11-04 16:50:44,053 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:50:44,053 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:50:44,063 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:50:45,395 INFO [INIT] Dropping privileges 2019-11-04 16:50:45,396 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:50:45,396 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:50:45,397 INFO [INIT] Marking all workers as offline 2019-11-04 16:50:45,423 INFO [INIT] Starting encryption 2019-11-04 16:50:45,424 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:50:45,424 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:50:45,425 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:50:45,439 INFO [INIT] LAVA master has started. 2019-11-04 16:50:45,439 INFO [INIT] Using protocol version 3 2019-11-04 16:50:54,104 DEBUG Dhanu => PING(20) 2019-11-04 16:50:54,105 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:50:59,839 DEBUG lava-logs => PING(20) 2019-11-04 16:51:05,445 INFO scheduling health checks: 2019-11-04 16:51:05,502 INFO scheduling jobs: 2019-11-04 16:51:05,504 DEBUG - kvm 2019-11-04 16:51:05,573 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:51:05,573 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:51:05,578 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:51:06,891 INFO [INIT] Dropping privileges 2019-11-04 16:51:06,891 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:51:06,892 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:51:06,893 INFO [INIT] Marking all workers as offline 2019-11-04 16:51:06,926 INFO [INIT] Starting encryption 2019-11-04 16:51:06,928 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:51:06,929 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:51:06,929 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:51:06,940 INFO [INIT] LAVA master has started. 2019-11-04 16:51:06,940 INFO [INIT] Using protocol version 3 2019-11-04 16:51:14,124 DEBUG Dhanu => PING(20) 2019-11-04 16:51:14,125 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:51:19,853 DEBUG lava-logs => PING(20) 2019-11-04 16:51:26,946 INFO scheduling health checks: 2019-11-04 16:51:27,027 INFO scheduling jobs: 2019-11-04 16:51:27,029 DEBUG - kvm 2019-11-04 16:51:27,109 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:51:27,109 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:51:27,114 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:51:28,516 INFO [INIT] Dropping privileges 2019-11-04 16:51:28,516 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:51:28,516 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:51:28,518 INFO [INIT] Marking all workers as offline 2019-11-04 16:51:28,572 INFO [INIT] Starting encryption 2019-11-04 16:51:28,574 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:51:28,574 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:51:28,575 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:51:28,587 INFO [INIT] LAVA master has started. 2019-11-04 16:51:28,588 INFO [INIT] Using protocol version 3 2019-11-04 16:51:34,143 DEBUG Dhanu => PING(20) 2019-11-04 16:51:34,143 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:51:39,866 DEBUG lava-logs => PING(20) 2019-11-04 16:51:48,597 INFO scheduling health checks: 2019-11-04 16:51:48,707 INFO scheduling jobs: 2019-11-04 16:51:48,709 DEBUG - kvm 2019-11-04 16:51:48,776 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:51:48,776 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:51:48,782 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:51:50,239 INFO [INIT] Dropping privileges 2019-11-04 16:51:50,239 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:51:50,240 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:51:50,241 INFO [INIT] Marking all workers as offline 2019-11-04 16:51:50,274 INFO [INIT] Starting encryption 2019-11-04 16:51:50,276 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:51:50,276 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:51:50,277 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:51:50,291 INFO [INIT] LAVA master has started. 2019-11-04 16:51:50,292 INFO [INIT] Using protocol version 3 2019-11-04 16:51:54,158 DEBUG Dhanu => PING(20) 2019-11-04 16:51:54,158 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:51:59,883 DEBUG lava-logs => PING(20) 2019-11-04 16:52:10,303 INFO scheduling health checks: 2019-11-04 16:52:10,365 INFO scheduling jobs: 2019-11-04 16:52:10,366 DEBUG - kvm 2019-11-04 16:52:10,447 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:52:10,447 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:52:10,452 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:52:11,590 INFO [INIT] Dropping privileges 2019-11-04 16:52:11,590 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:52:11,590 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:52:11,592 INFO [INIT] Marking all workers as offline 2019-11-04 16:52:11,632 INFO [INIT] Starting encryption 2019-11-04 16:52:11,633 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:52:11,634 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:52:11,634 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:52:11,649 INFO [INIT] LAVA master has started. 2019-11-04 16:52:11,650 INFO [INIT] Using protocol version 3 2019-11-04 16:52:14,175 DEBUG Dhanu => PING(20) 2019-11-04 16:52:14,176 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:52:19,904 DEBUG lava-logs => PING(20) 2019-11-04 16:52:31,662 INFO scheduling health checks: 2019-11-04 16:52:31,722 INFO scheduling jobs: 2019-11-04 16:52:31,724 DEBUG - kvm 2019-11-04 16:52:31,798 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:52:31,799 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:52:31,803 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:52:33,241 INFO [INIT] Dropping privileges 2019-11-04 16:52:33,241 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:52:33,243 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:52:33,245 INFO [INIT] Marking all workers as offline 2019-11-04 16:52:33,279 INFO [INIT] Starting encryption 2019-11-04 16:52:33,281 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:52:33,281 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:52:33,282 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:52:33,295 INFO [INIT] LAVA master has started. 2019-11-04 16:52:33,295 INFO [INIT] Using protocol version 3 2019-11-04 16:52:34,194 DEBUG Dhanu => PING(20) 2019-11-04 16:52:34,194 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:52:39,922 DEBUG lava-logs => PING(20) 2019-11-04 16:52:53,309 INFO scheduling health checks: 2019-11-04 16:52:53,391 INFO scheduling jobs: 2019-11-04 16:52:53,393 DEBUG - kvm 2019-11-04 16:52:53,468 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:52:53,469 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:52:53,473 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:52:54,691 INFO [INIT] Dropping privileges 2019-11-04 16:52:54,691 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:52:54,692 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:52:54,693 INFO [INIT] Marking all workers as offline 2019-11-04 16:52:54,726 INFO [INIT] Starting encryption 2019-11-04 16:52:54,727 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:52:54,728 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:52:54,728 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:52:54,743 INFO [INIT] LAVA master has started. 2019-11-04 16:52:54,743 INFO [INIT] Using protocol version 3 2019-11-04 16:52:54,772 DEBUG Dhanu => PING(20) 2019-11-04 16:52:54,772 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:52:59,934 DEBUG lava-logs => PING(20) 2019-11-04 16:53:14,754 INFO scheduling health checks: 2019-11-04 16:53:14,804 INFO scheduling jobs: 2019-11-04 16:53:14,807 DEBUG - kvm 2019-11-04 16:53:14,884 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:53:14,884 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:53:14,888 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:53:16,207 INFO [INIT] Dropping privileges 2019-11-04 16:53:16,207 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:53:16,207 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:53:16,210 INFO [INIT] Marking all workers as offline 2019-11-04 16:53:16,252 INFO [INIT] Starting encryption 2019-11-04 16:53:16,253 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:53:16,253 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:53:16,253 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:53:16,264 INFO [INIT] LAVA master has started. 2019-11-04 16:53:16,264 INFO [INIT] Using protocol version 3 2019-11-04 16:53:19,951 DEBUG lava-logs => PING(20) 2019-11-04 16:53:34,802 DEBUG Dhanu => PING(20) 2019-11-04 16:53:34,802 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:53:36,266 INFO scheduling health checks: 2019-11-04 16:53:36,362 INFO scheduling jobs: 2019-11-04 16:53:36,363 DEBUG - kvm 2019-11-04 16:53:36,458 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:53:36,459 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:53:36,464 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:53:37,942 INFO [INIT] Dropping privileges 2019-11-04 16:53:37,943 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:53:37,943 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:53:37,947 INFO [INIT] Marking all workers as offline 2019-11-04 16:53:38,013 INFO [INIT] Starting encryption 2019-11-04 16:53:38,014 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:53:38,015 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:53:38,015 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:53:38,025 INFO [INIT] LAVA master has started. 2019-11-04 16:53:38,026 INFO [INIT] Using protocol version 3 2019-11-04 16:53:39,964 DEBUG lava-logs => PING(20) 2019-11-04 16:53:54,818 DEBUG Dhanu => PING(20) 2019-11-04 16:53:54,818 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:53:58,029 INFO scheduling health checks: 2019-11-04 16:53:58,075 INFO scheduling jobs: 2019-11-04 16:53:58,077 DEBUG - kvm 2019-11-04 16:53:58,153 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:53:58,153 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:53:58,158 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:53:59,558 INFO [INIT] Dropping privileges 2019-11-04 16:53:59,559 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:53:59,559 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:53:59,561 INFO [INIT] Marking all workers as offline 2019-11-04 16:53:59,609 INFO [INIT] Starting encryption 2019-11-04 16:53:59,611 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:53:59,611 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:53:59,611 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:53:59,623 INFO [INIT] LAVA master has started. 2019-11-04 16:53:59,624 INFO [INIT] Using protocol version 3 2019-11-04 16:53:59,975 DEBUG lava-logs => PING(20) 2019-11-04 16:54:14,832 DEBUG Dhanu => PING(20) 2019-11-04 16:54:14,832 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:54:19,626 INFO scheduling health checks: 2019-11-04 16:54:19,700 INFO scheduling jobs: 2019-11-04 16:54:19,702 DEBUG - kvm 2019-11-04 16:54:19,812 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:54:19,812 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:54:19,819 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:54:21,320 INFO [INIT] Dropping privileges 2019-11-04 16:54:21,320 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:54:21,320 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:54:21,322 INFO [INIT] Marking all workers as offline 2019-11-04 16:54:21,356 INFO [INIT] Starting encryption 2019-11-04 16:54:21,358 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:54:21,358 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:54:21,359 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:54:21,370 INFO [INIT] LAVA master has started. 2019-11-04 16:54:21,370 INFO [INIT] Using protocol version 3 2019-11-04 16:54:21,427 DEBUG lava-logs => PING(20) 2019-11-04 16:54:34,846 DEBUG Dhanu => PING(20) 2019-11-04 16:54:34,847 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:54:41,377 INFO scheduling health checks: 2019-11-04 16:54:41,424 INFO scheduling jobs: 2019-11-04 16:54:41,425 DEBUG - kvm 2019-11-04 16:54:41,496 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:54:41,496 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:54:41,509 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:54:42,977 INFO [INIT] Dropping privileges 2019-11-04 16:54:42,978 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:54:42,978 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:54:42,980 INFO [INIT] Marking all workers as offline 2019-11-04 16:54:43,014 INFO [INIT] Starting encryption 2019-11-04 16:54:43,015 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:54:43,015 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:54:43,016 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:54:43,028 INFO [INIT] LAVA master has started. 2019-11-04 16:54:43,028 INFO [INIT] Using protocol version 3 2019-11-04 16:54:54,858 DEBUG Dhanu => PING(20) 2019-11-04 16:54:54,859 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:55:01,458 DEBUG lava-logs => PING(20) 2019-11-04 16:55:03,030 INFO scheduling health checks: 2019-11-04 16:55:03,070 INFO scheduling jobs: 2019-11-04 16:55:03,072 DEBUG - kvm 2019-11-04 16:55:03,163 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:55:03,163 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:55:03,170 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:55:04,491 INFO [INIT] Dropping privileges 2019-11-04 16:55:04,491 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:55:04,491 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:55:04,493 INFO [INIT] Marking all workers as offline 2019-11-04 16:55:04,515 INFO [INIT] Starting encryption 2019-11-04 16:55:04,516 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:55:04,517 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:55:04,517 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:55:04,524 INFO [INIT] LAVA master has started. 2019-11-04 16:55:04,525 INFO [INIT] Using protocol version 3 2019-11-04 16:55:14,870 DEBUG Dhanu => PING(20) 2019-11-04 16:55:14,870 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:55:21,476 DEBUG lava-logs => PING(20) 2019-11-04 16:55:24,528 INFO scheduling health checks: 2019-11-04 16:55:24,573 INFO scheduling jobs: 2019-11-04 16:55:24,574 DEBUG - kvm 2019-11-04 16:55:24,651 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:55:24,652 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:55:24,656 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:55:25,984 INFO [INIT] Dropping privileges 2019-11-04 16:55:25,984 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:55:25,984 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:55:25,986 INFO [INIT] Marking all workers as offline 2019-11-04 16:55:26,029 INFO [INIT] Starting encryption 2019-11-04 16:55:26,031 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:55:26,031 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:55:26,031 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:55:26,041 INFO [INIT] LAVA master has started. 2019-11-04 16:55:26,041 INFO [INIT] Using protocol version 3 2019-11-04 16:55:34,886 DEBUG Dhanu => PING(20) 2019-11-04 16:55:34,887 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:55:41,487 DEBUG lava-logs => PING(20) 2019-11-04 16:55:46,042 INFO scheduling health checks: 2019-11-04 16:55:46,086 INFO scheduling jobs: 2019-11-04 16:55:46,087 DEBUG - kvm 2019-11-04 16:55:46,166 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:55:46,166 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:55:46,171 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:55:47,460 INFO [INIT] Dropping privileges 2019-11-04 16:55:47,461 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:55:47,461 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:55:47,462 INFO [INIT] Marking all workers as offline 2019-11-04 16:55:47,498 INFO [INIT] Starting encryption 2019-11-04 16:55:47,499 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:55:47,500 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:55:47,500 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:55:47,512 INFO [INIT] LAVA master has started. 2019-11-04 16:55:47,512 INFO [INIT] Using protocol version 3 2019-11-04 16:55:54,902 DEBUG Dhanu => PING(20) 2019-11-04 16:55:54,903 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:56:01,498 DEBUG lava-logs => PING(20) 2019-11-04 16:56:07,518 INFO scheduling health checks: 2019-11-04 16:56:07,621 INFO scheduling jobs: 2019-11-04 16:56:07,623 DEBUG - kvm 2019-11-04 16:56:07,696 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:56:07,696 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:56:07,703 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:56:09,002 INFO [INIT] Dropping privileges 2019-11-04 16:56:09,003 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:56:09,003 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:56:09,005 INFO [INIT] Marking all workers as offline 2019-11-04 16:56:09,045 INFO [INIT] Starting encryption 2019-11-04 16:56:09,046 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:56:09,046 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:56:09,047 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:56:09,062 INFO [INIT] LAVA master has started. 2019-11-04 16:56:09,062 INFO [INIT] Using protocol version 3 2019-11-04 16:56:14,918 DEBUG Dhanu => PING(20) 2019-11-04 16:56:14,919 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:56:21,511 DEBUG lava-logs => PING(20) 2019-11-04 16:56:29,070 INFO scheduling health checks: 2019-11-04 16:56:29,101 INFO scheduling jobs: 2019-11-04 16:56:29,103 DEBUG - kvm 2019-11-04 16:56:29,182 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:56:29,182 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:56:29,190 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:56:30,429 INFO [INIT] Dropping privileges 2019-11-04 16:56:30,429 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:56:30,430 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:56:30,432 INFO [INIT] Marking all workers as offline 2019-11-04 16:56:30,469 INFO [INIT] Starting encryption 2019-11-04 16:56:30,470 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:56:30,470 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:56:30,471 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:56:30,484 INFO [INIT] LAVA master has started. 2019-11-04 16:56:30,484 INFO [INIT] Using protocol version 3 2019-11-04 16:56:34,938 DEBUG Dhanu => PING(20) 2019-11-04 16:56:34,939 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:56:41,522 DEBUG lava-logs => PING(20) 2019-11-04 16:56:50,493 INFO scheduling health checks: 2019-11-04 16:56:50,526 INFO scheduling jobs: 2019-11-04 16:56:50,527 DEBUG - kvm 2019-11-04 16:56:50,615 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:56:50,615 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:56:50,620 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:56:51,942 INFO [INIT] Dropping privileges 2019-11-04 16:56:51,943 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:56:51,943 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:56:51,944 INFO [INIT] Marking all workers as offline 2019-11-04 16:56:51,982 INFO [INIT] Starting encryption 2019-11-04 16:56:51,983 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:56:51,983 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:56:51,983 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:56:51,995 INFO [INIT] LAVA master has started. 2019-11-04 16:56:51,995 INFO [INIT] Using protocol version 3 2019-11-04 16:56:54,954 DEBUG Dhanu => PING(20) 2019-11-04 16:56:54,954 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:57:01,542 DEBUG lava-logs => PING(20) 2019-11-04 16:57:12,006 INFO scheduling health checks: 2019-11-04 16:57:12,039 INFO scheduling jobs: 2019-11-04 16:57:12,042 DEBUG - kvm 2019-11-04 16:57:12,128 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:57:12,128 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:57:12,134 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:57:13,494 INFO [INIT] Dropping privileges 2019-11-04 16:57:13,494 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:57:13,495 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:57:13,496 INFO [INIT] Marking all workers as offline 2019-11-04 16:57:13,530 INFO [INIT] Starting encryption 2019-11-04 16:57:13,531 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:57:13,532 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:57:13,532 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:57:13,544 INFO [INIT] LAVA master has started. 2019-11-04 16:57:13,545 INFO [INIT] Using protocol version 3 2019-11-04 16:57:14,966 DEBUG Dhanu => PING(20) 2019-11-04 16:57:14,966 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:57:21,558 DEBUG lava-logs => PING(20) 2019-11-04 16:57:33,554 INFO scheduling health checks: 2019-11-04 16:57:33,608 INFO scheduling jobs: 2019-11-04 16:57:33,609 DEBUG - kvm 2019-11-04 16:57:33,698 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:57:33,698 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:57:33,707 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:57:35,244 INFO [INIT] Dropping privileges 2019-11-04 16:57:35,244 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:57:35,244 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:57:35,246 INFO [INIT] Marking all workers as offline 2019-11-04 16:57:35,286 INFO [INIT] Starting encryption 2019-11-04 16:57:35,289 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:57:35,289 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:57:35,289 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:57:35,301 INFO [INIT] LAVA master has started. 2019-11-04 16:57:35,301 INFO [INIT] Using protocol version 3 2019-11-04 16:57:35,451 DEBUG Dhanu => PING(20) 2019-11-04 16:57:35,451 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:57:41,571 DEBUG lava-logs => PING(20) 2019-11-04 16:57:55,314 INFO scheduling health checks: 2019-11-04 16:57:55,365 INFO scheduling jobs: 2019-11-04 16:57:55,370 DEBUG - kvm 2019-11-04 16:57:55,450 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:57:55,450 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:57:55,455 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:57:56,813 INFO [INIT] Dropping privileges 2019-11-04 16:57:56,813 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:57:56,814 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:57:56,816 INFO [INIT] Marking all workers as offline 2019-11-04 16:57:56,844 INFO [INIT] Starting encryption 2019-11-04 16:57:56,845 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:57:56,845 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:57:56,845 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:57:56,855 INFO [INIT] LAVA master has started. 2019-11-04 16:57:56,857 INFO [INIT] Using protocol version 3 2019-11-04 16:57:56,938 DEBUG Dhanu => PING(20) 2019-11-04 16:57:56,939 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:58:01,586 DEBUG lava-logs => PING(20) 2019-11-04 16:58:16,869 INFO scheduling health checks: 2019-11-04 16:58:16,934 INFO scheduling jobs: 2019-11-04 16:58:16,938 DEBUG - kvm 2019-11-04 16:58:17,042 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:58:17,042 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:58:17,049 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:58:18,527 INFO [INIT] Dropping privileges 2019-11-04 16:58:18,528 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:58:18,528 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:58:18,530 INFO [INIT] Marking all workers as offline 2019-11-04 16:58:18,568 INFO [INIT] Starting encryption 2019-11-04 16:58:18,569 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:58:18,569 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:58:18,569 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:58:18,581 INFO [INIT] LAVA master has started. 2019-11-04 16:58:18,581 INFO [INIT] Using protocol version 3 2019-11-04 16:58:21,598 DEBUG lava-logs => PING(20) 2019-11-04 16:58:36,974 DEBUG Dhanu => PING(20) 2019-11-04 16:58:36,974 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:58:38,583 INFO scheduling health checks: 2019-11-04 16:58:38,614 INFO scheduling jobs: 2019-11-04 16:58:38,616 DEBUG - kvm 2019-11-04 16:58:38,690 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:58:38,690 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:58:38,695 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:58:39,857 INFO [INIT] Dropping privileges 2019-11-04 16:58:39,857 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:58:39,858 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:58:39,859 INFO [INIT] Marking all workers as offline 2019-11-04 16:58:39,893 INFO [INIT] Starting encryption 2019-11-04 16:58:39,896 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:58:39,896 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:58:39,897 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:58:39,915 INFO [INIT] LAVA master has started. 2019-11-04 16:58:39,916 INFO [INIT] Using protocol version 3 2019-11-04 16:58:41,612 DEBUG lava-logs => PING(20) 2019-11-04 16:58:56,986 DEBUG Dhanu => PING(20) 2019-11-04 16:58:56,986 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:58:59,918 INFO scheduling health checks: 2019-11-04 16:58:59,972 INFO scheduling jobs: 2019-11-04 16:58:59,973 DEBUG - kvm 2019-11-04 16:59:00,052 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:59:00,053 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:59:00,061 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:59:02,380 INFO [INIT] Dropping privileges 2019-11-04 16:59:02,424 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:59:02,425 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:59:02,428 INFO [INIT] Marking all workers as offline 2019-11-04 16:59:02,483 INFO [INIT] Starting encryption 2019-11-04 16:59:02,484 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:59:02,485 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:59:02,485 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:59:02,495 INFO [INIT] LAVA master has started. 2019-11-04 16:59:02,495 INFO [INIT] Using protocol version 3 2019-11-04 16:59:02,592 DEBUG lava-logs => PING(20) 2019-11-04 16:59:17,003 DEBUG Dhanu => PING(20) 2019-11-04 16:59:17,003 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:59:22,500 INFO scheduling health checks: 2019-11-04 16:59:22,541 INFO scheduling jobs: 2019-11-04 16:59:22,542 DEBUG - kvm 2019-11-04 16:59:22,610 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:59:22,611 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:59:22,615 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:59:25,148 INFO [INIT] Dropping privileges 2019-11-04 16:59:25,148 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:59:25,149 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:59:25,152 INFO [INIT] Marking all workers as offline 2019-11-04 16:59:25,187 INFO [INIT] Starting encryption 2019-11-04 16:59:25,189 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:59:25,189 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:59:25,190 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:59:25,205 INFO [INIT] LAVA master has started. 2019-11-04 16:59:25,206 INFO [INIT] Using protocol version 3 2019-11-04 16:59:37,024 DEBUG Dhanu => PING(20) 2019-11-04 16:59:37,042 WARNING Unknown dispatcher (server crashed) 2019-11-04 16:59:42,629 DEBUG lava-logs => PING(20) 2019-11-04 16:59:45,208 INFO scheduling health checks: 2019-11-04 16:59:45,254 INFO scheduling jobs: 2019-11-04 16:59:45,256 DEBUG - kvm 2019-11-04 16:59:45,330 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 16:59:45,330 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 16:59:45,334 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 16:59:47,906 INFO [INIT] Dropping privileges 2019-11-04 16:59:47,907 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 16:59:47,907 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 16:59:47,909 INFO [INIT] Marking all workers as offline 2019-11-04 16:59:47,933 INFO [INIT] Starting encryption 2019-11-04 16:59:47,934 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 16:59:47,934 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 16:59:47,935 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 16:59:47,944 INFO [INIT] LAVA master has started. 2019-11-04 16:59:47,945 INFO [INIT] Using protocol version 3 2019-11-04 16:59:57,060 DEBUG Dhanu => PING(20) 2019-11-04 16:59:57,061 WARNING Unknown dispatcher (server crashed) 2019-11-04 17:00:02,642 DEBUG lava-logs => PING(20) 2019-11-04 17:00:07,950 INFO scheduling health checks: 2019-11-04 17:00:08,001 INFO scheduling jobs: 2019-11-04 17:00:08,003 DEBUG - kvm 2019-11-04 17:00:08,074 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 17:00:08,074 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 17:00:08,079 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 17:00:09,372 INFO [INIT] Dropping privileges 2019-11-04 17:00:09,373 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 17:00:09,373 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 17:00:09,375 INFO [INIT] Marking all workers as offline 2019-11-04 17:00:09,413 INFO [INIT] Starting encryption 2019-11-04 17:00:09,415 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 17:00:09,415 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 17:00:09,416 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 17:00:09,428 INFO [INIT] LAVA master has started. 2019-11-04 17:00:09,429 INFO [INIT] Using protocol version 3 2019-11-04 17:00:17,076 DEBUG Dhanu => PING(20) 2019-11-04 17:00:17,077 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:28:21,038 DEBUG lava-logs => PING(20) 2019-11-04 11:28:21,074 DEBUG lava-logs => PING(20) 2019-11-04 11:28:25,846 INFO [POLL] Received a signal, leaving 2019-11-04 11:28:25,846 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:28:26,826 INFO [INIT] Dropping privileges 2019-11-04 11:28:26,826 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:28:26,826 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:28:26,828 INFO [INIT] Marking all workers as offline 2019-11-04 11:28:26,887 INFO [INIT] Starting encryption 2019-11-04 11:28:26,889 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:28:26,889 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:28:26,890 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:28:26,903 INFO [INIT] LAVA master has started. 2019-11-04 11:28:26,903 INFO [INIT] Using protocol version 3 2019-11-04 11:28:34,115 INFO Dhanu => HELLO 2019-11-04 11:28:34,116 WARNING New dispatcher 2019-11-04 11:28:39,122 DEBUG Dhanu => PING(20) 2019-11-04 11:28:41,092 DEBUG lava-logs => PING(20) 2019-11-04 11:28:46,904 INFO scheduling health checks: 2019-11-04 11:28:46,966 INFO scheduling jobs: 2019-11-04 11:28:46,968 DEBUG - kvm 2019-11-04 11:28:47,051 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:28:47,051 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:28:47,055 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:28:48,156 INFO [INIT] Dropping privileges 2019-11-04 11:28:48,156 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:28:48,156 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:28:48,158 INFO [INIT] Marking all workers as offline 2019-11-04 11:28:48,189 INFO [INIT] Starting encryption 2019-11-04 11:28:48,191 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:28:48,191 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:28:48,192 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:28:48,211 INFO [INIT] LAVA master has started. 2019-11-04 11:28:48,211 INFO [INIT] Using protocol version 3 2019-11-04 11:28:59,139 DEBUG Dhanu => PING(20) 2019-11-04 11:28:59,140 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:29:01,107 DEBUG lava-logs => PING(20) 2019-11-04 11:29:08,217 INFO scheduling health checks: 2019-11-04 11:29:08,268 INFO scheduling jobs: 2019-11-04 11:29:08,270 DEBUG - kvm 2019-11-04 11:29:08,344 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:29:08,345 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:29:08,350 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:29:10,304 INFO [INIT] Dropping privileges 2019-11-04 11:29:10,304 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:29:10,304 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:29:10,307 INFO [INIT] Marking all workers as offline 2019-11-04 11:29:10,369 INFO [INIT] Starting encryption 2019-11-04 11:29:10,369 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:29:10,370 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:29:10,370 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:29:10,382 INFO [INIT] LAVA master has started. 2019-11-04 11:29:10,382 INFO [INIT] Using protocol version 3 2019-11-04 11:29:19,157 DEBUG Dhanu => PING(20) 2019-11-04 11:29:19,157 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:29:21,117 DEBUG lava-logs => PING(20) 2019-11-04 11:29:30,388 INFO scheduling health checks: 2019-11-04 11:29:30,436 INFO scheduling jobs: 2019-11-04 11:29:30,438 DEBUG - kvm 2019-11-04 11:29:30,521 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:29:30,521 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:29:30,529 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:29:31,730 INFO [INIT] Dropping privileges 2019-11-04 11:29:31,730 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:29:31,730 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:29:31,732 INFO [INIT] Marking all workers as offline 2019-11-04 11:29:31,771 INFO [INIT] Starting encryption 2019-11-04 11:29:31,773 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:29:31,773 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:29:31,773 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:29:31,786 INFO [INIT] LAVA master has started. 2019-11-04 11:29:31,787 INFO [INIT] Using protocol version 3 2019-11-04 11:29:39,174 DEBUG Dhanu => PING(20) 2019-11-04 11:29:39,174 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:29:41,134 DEBUG lava-logs => PING(20) 2019-11-04 11:29:51,798 INFO scheduling health checks: 2019-11-04 11:29:51,861 INFO scheduling jobs: 2019-11-04 11:29:51,863 DEBUG - kvm 2019-11-04 11:29:51,937 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:29:51,937 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:29:51,943 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:29:53,153 INFO [INIT] Dropping privileges 2019-11-04 11:29:53,153 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:29:53,153 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:29:53,155 INFO [INIT] Marking all workers as offline 2019-11-04 11:29:53,184 INFO [INIT] Starting encryption 2019-11-04 11:29:53,185 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:29:53,185 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:29:53,185 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:29:53,200 INFO [INIT] LAVA master has started. 2019-11-04 11:29:53,200 INFO [INIT] Using protocol version 3 2019-11-04 11:29:59,195 DEBUG Dhanu => PING(20) 2019-11-04 11:29:59,195 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:30:01,152 DEBUG lava-logs => PING(20) 2019-11-04 11:30:13,212 INFO scheduling health checks: 2019-11-04 11:30:13,319 INFO scheduling jobs: 2019-11-04 11:30:13,321 DEBUG - kvm 2019-11-04 11:30:13,396 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:30:13,396 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:30:13,400 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:30:14,677 INFO [INIT] Dropping privileges 2019-11-04 11:30:14,677 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:30:14,678 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:30:14,679 INFO [INIT] Marking all workers as offline 2019-11-04 11:30:14,708 INFO [INIT] Starting encryption 2019-11-04 11:30:14,709 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:30:14,710 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:30:14,710 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:30:14,723 INFO [INIT] LAVA master has started. 2019-11-04 11:30:14,724 INFO [INIT] Using protocol version 3 2019-11-04 11:30:19,214 DEBUG Dhanu => PING(20) 2019-11-04 11:30:19,214 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:30:21,171 DEBUG lava-logs => PING(20) 2019-11-04 11:30:34,737 INFO scheduling health checks: 2019-11-04 11:30:34,799 INFO scheduling jobs: 2019-11-04 11:30:34,800 DEBUG - kvm 2019-11-04 11:30:34,876 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:30:34,876 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:30:34,881 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:30:36,183 INFO [INIT] Dropping privileges 2019-11-04 11:30:36,183 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:30:36,184 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:30:36,185 INFO [INIT] Marking all workers as offline 2019-11-04 11:30:36,210 INFO [INIT] Starting encryption 2019-11-04 11:30:36,212 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:30:36,212 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:30:36,212 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:30:36,226 INFO [INIT] LAVA master has started. 2019-11-04 11:30:36,226 INFO [INIT] Using protocol version 3 2019-11-04 11:30:39,229 DEBUG Dhanu => PING(20) 2019-11-04 11:30:39,229 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:30:41,189 DEBUG lava-logs => PING(20) 2019-11-04 11:30:56,242 INFO scheduling health checks: 2019-11-04 11:30:56,301 INFO scheduling jobs: 2019-11-04 11:30:56,303 DEBUG - kvm 2019-11-04 11:30:56,376 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:30:56,376 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:30:56,380 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:30:57,647 INFO [INIT] Dropping privileges 2019-11-04 11:30:57,647 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:30:57,648 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:30:57,649 INFO [INIT] Marking all workers as offline 2019-11-04 11:30:57,680 INFO [INIT] Starting encryption 2019-11-04 11:30:57,682 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:30:57,682 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:30:57,683 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:30:57,698 INFO [INIT] LAVA master has started. 2019-11-04 11:30:57,699 INFO [INIT] Using protocol version 3 2019-11-04 11:30:59,249 DEBUG Dhanu => PING(20) 2019-11-04 11:30:59,250 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:31:01,204 DEBUG lava-logs => PING(20) 2019-11-04 11:31:17,715 INFO scheduling health checks: 2019-11-04 11:31:17,836 INFO scheduling jobs: 2019-11-04 11:31:17,838 DEBUG - kvm 2019-11-04 11:31:17,912 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:31:17,913 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:31:17,917 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:31:19,148 INFO [INIT] Dropping privileges 2019-11-04 11:31:19,148 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:31:19,149 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:31:19,150 INFO [INIT] Marking all workers as offline 2019-11-04 11:31:19,181 INFO [INIT] Starting encryption 2019-11-04 11:31:19,182 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:31:19,182 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:31:19,183 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:31:19,196 INFO [INIT] LAVA master has started. 2019-11-04 11:31:19,196 INFO [INIT] Using protocol version 3 2019-11-04 11:31:19,268 DEBUG Dhanu => PING(20) 2019-11-04 11:31:19,269 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:31:21,216 DEBUG lava-logs => PING(20) 2019-11-04 11:31:39,214 INFO scheduling health checks: 2019-11-04 11:31:39,271 INFO scheduling jobs: 2019-11-04 11:31:39,273 DEBUG - kvm 2019-11-04 11:31:39,361 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:31:39,362 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:31:39,370 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:31:40,712 INFO [INIT] Dropping privileges 2019-11-04 11:31:40,712 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:31:40,713 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:31:40,715 INFO [INIT] Marking all workers as offline 2019-11-04 11:31:40,739 INFO [INIT] Starting encryption 2019-11-04 11:31:40,741 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:31:40,741 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:31:40,741 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:31:40,755 INFO [INIT] LAVA master has started. 2019-11-04 11:31:40,756 INFO [INIT] Using protocol version 3 2019-11-04 11:31:41,228 DEBUG lava-logs => PING(20) 2019-11-04 11:31:59,302 DEBUG Dhanu => PING(20) 2019-11-04 11:31:59,302 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:32:00,758 INFO scheduling health checks: 2019-11-04 11:32:00,807 INFO scheduling jobs: 2019-11-04 11:32:00,809 DEBUG - kvm 2019-11-04 11:32:00,885 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:32:00,886 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:32:00,895 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:32:02,414 INFO [INIT] Dropping privileges 2019-11-04 11:32:02,414 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:32:02,415 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:32:02,417 INFO [INIT] Marking all workers as offline 2019-11-04 11:32:02,441 INFO [INIT] Starting encryption 2019-11-04 11:32:02,442 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:32:02,442 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:32:02,443 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:32:02,458 INFO [INIT] LAVA master has started. 2019-11-04 11:32:02,458 INFO [INIT] Using protocol version 3 2019-11-04 11:32:02,547 DEBUG lava-logs => PING(20) 2019-11-04 11:32:19,317 DEBUG Dhanu => PING(20) 2019-11-04 11:32:19,317 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:32:22,461 INFO scheduling health checks: 2019-11-04 11:32:22,487 INFO scheduling jobs: 2019-11-04 11:32:22,490 DEBUG - kvm 2019-11-04 11:32:22,570 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:32:22,570 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:32:22,575 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:32:24,133 INFO [INIT] Dropping privileges 2019-11-04 11:32:24,133 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:32:24,134 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:32:24,135 INFO [INIT] Marking all workers as offline 2019-11-04 11:32:24,199 INFO [INIT] Starting encryption 2019-11-04 11:32:24,201 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:32:24,202 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:32:24,202 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:32:24,219 INFO [INIT] LAVA master has started. 2019-11-04 11:32:24,219 INFO [INIT] Using protocol version 3 2019-11-04 11:32:39,338 DEBUG Dhanu => PING(20) 2019-11-04 11:32:39,338 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:32:42,581 DEBUG lava-logs => PING(20) 2019-11-04 11:32:44,220 INFO scheduling health checks: 2019-11-04 11:32:44,322 INFO scheduling jobs: 2019-11-04 11:32:44,323 DEBUG - kvm 2019-11-04 11:32:44,405 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:32:44,405 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:32:44,411 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:32:45,948 INFO [INIT] Dropping privileges 2019-11-04 11:32:45,949 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:32:45,949 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:32:45,951 INFO [INIT] Marking all workers as offline 2019-11-04 11:32:45,989 INFO [INIT] Starting encryption 2019-11-04 11:32:45,990 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:32:45,990 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:32:45,990 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:32:46,000 INFO [INIT] LAVA master has started. 2019-11-04 11:32:46,000 INFO [INIT] Using protocol version 3 2019-11-04 11:32:59,353 DEBUG Dhanu => PING(20) 2019-11-04 11:32:59,353 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:33:02,593 DEBUG lava-logs => PING(20) 2019-11-04 11:33:06,004 INFO scheduling health checks: 2019-11-04 11:33:06,069 INFO scheduling jobs: 2019-11-04 11:33:06,070 DEBUG - kvm 2019-11-04 11:33:06,151 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:33:06,151 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:33:06,157 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:33:07,798 INFO [INIT] Dropping privileges 2019-11-04 11:33:07,798 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:33:07,798 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:33:07,801 INFO [INIT] Marking all workers as offline 2019-11-04 11:33:07,825 INFO [INIT] Starting encryption 2019-11-04 11:33:07,825 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:33:07,825 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:33:07,826 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:33:07,835 INFO [INIT] LAVA master has started. 2019-11-04 11:33:07,836 INFO [INIT] Using protocol version 3 2019-11-04 11:33:19,365 DEBUG Dhanu => PING(20) 2019-11-04 11:33:19,365 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:33:22,605 DEBUG lava-logs => PING(20) 2019-11-04 11:33:27,840 INFO scheduling health checks: 2019-11-04 11:33:27,916 INFO scheduling jobs: 2019-11-04 11:33:27,917 DEBUG - kvm 2019-11-04 11:33:27,998 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:33:27,998 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:33:28,006 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:33:29,286 INFO [INIT] Dropping privileges 2019-11-04 11:33:29,286 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:33:29,286 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:33:29,288 INFO [INIT] Marking all workers as offline 2019-11-04 11:33:29,316 INFO [INIT] Starting encryption 2019-11-04 11:33:29,317 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:33:29,317 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:33:29,317 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:33:29,326 INFO [INIT] LAVA master has started. 2019-11-04 11:33:29,327 INFO [INIT] Using protocol version 3 2019-11-04 11:33:39,381 DEBUG Dhanu => PING(20) 2019-11-04 11:33:39,381 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:33:42,617 DEBUG lava-logs => PING(20) 2019-11-04 11:33:49,333 INFO scheduling health checks: 2019-11-04 11:33:49,373 INFO scheduling jobs: 2019-11-04 11:33:49,375 DEBUG - kvm 2019-11-04 11:33:49,462 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:33:49,462 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:33:49,467 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:33:50,756 INFO [INIT] Dropping privileges 2019-11-04 11:33:50,756 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:33:50,756 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:33:50,759 INFO [INIT] Marking all workers as offline 2019-11-04 11:33:50,796 INFO [INIT] Starting encryption 2019-11-04 11:33:50,799 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:33:50,799 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:33:50,800 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:33:50,817 INFO [INIT] LAVA master has started. 2019-11-04 11:33:50,817 INFO [INIT] Using protocol version 3 2019-11-04 11:33:59,394 DEBUG Dhanu => PING(20) 2019-11-04 11:33:59,395 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:34:02,629 DEBUG lava-logs => PING(20) 2019-11-04 11:34:10,824 INFO scheduling health checks: 2019-11-04 11:34:10,865 INFO scheduling jobs: 2019-11-04 11:34:10,866 DEBUG - kvm 2019-11-04 11:34:10,957 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:34:10,957 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:34:10,969 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:34:12,263 INFO [INIT] Dropping privileges 2019-11-04 11:34:12,264 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:34:12,264 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:34:12,266 INFO [INIT] Marking all workers as offline 2019-11-04 11:34:12,287 INFO [INIT] Starting encryption 2019-11-04 11:34:12,288 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:34:12,288 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:34:12,288 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:34:12,300 INFO [INIT] LAVA master has started. 2019-11-04 11:34:12,301 INFO [INIT] Using protocol version 3 2019-11-04 11:34:19,405 DEBUG Dhanu => PING(20) 2019-11-04 11:34:19,405 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:34:22,637 DEBUG lava-logs => PING(20) 2019-11-04 11:34:32,308 INFO scheduling health checks: 2019-11-04 11:34:32,344 INFO scheduling jobs: 2019-11-04 11:34:32,346 DEBUG - kvm 2019-11-04 11:34:32,420 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:34:32,421 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:34:32,429 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:34:34,090 INFO [INIT] Dropping privileges 2019-11-04 11:34:34,090 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:34:34,090 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:34:34,123 INFO [INIT] Marking all workers as offline 2019-11-04 11:34:34,200 INFO [INIT] Starting encryption 2019-11-04 11:34:34,201 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:34:34,201 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:34:34,201 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:34:34,210 INFO [INIT] LAVA master has started. 2019-11-04 11:34:34,210 INFO [INIT] Using protocol version 3 2019-11-04 11:34:39,417 DEBUG Dhanu => PING(20) 2019-11-04 11:34:39,417 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:34:42,653 DEBUG lava-logs => PING(20) 2019-11-04 11:34:54,220 INFO scheduling health checks: 2019-11-04 11:34:54,280 INFO scheduling jobs: 2019-11-04 11:34:54,283 DEBUG - kvm 2019-11-04 11:34:54,366 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:34:54,366 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:34:54,371 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:34:55,814 INFO [INIT] Dropping privileges 2019-11-04 11:34:55,814 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:34:55,814 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:34:55,816 INFO [INIT] Marking all workers as offline 2019-11-04 11:34:55,849 INFO [INIT] Starting encryption 2019-11-04 11:34:55,850 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:34:55,851 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:34:55,851 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:34:55,866 INFO [INIT] LAVA master has started. 2019-11-04 11:34:55,867 INFO [INIT] Using protocol version 3 2019-11-04 11:34:59,433 DEBUG Dhanu => PING(20) 2019-11-04 11:34:59,434 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:35:02,669 DEBUG lava-logs => PING(20) 2019-11-04 11:35:15,878 INFO scheduling health checks: 2019-11-04 11:35:15,915 INFO scheduling jobs: 2019-11-04 11:35:15,916 DEBUG - kvm 2019-11-04 11:35:15,987 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:35:15,987 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:35:15,992 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:35:18,579 INFO [INIT] Dropping privileges 2019-11-04 11:35:18,579 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:35:18,579 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:35:18,582 INFO [INIT] Marking all workers as offline 2019-11-04 11:35:18,616 INFO [INIT] Starting encryption 2019-11-04 11:35:18,617 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:35:18,617 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:35:18,617 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:35:18,635 INFO [INIT] LAVA master has started. 2019-11-04 11:35:18,635 INFO [INIT] Using protocol version 3 2019-11-04 11:35:19,450 DEBUG Dhanu => PING(20) 2019-11-04 11:35:19,451 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:35:22,686 DEBUG lava-logs => PING(20) 2019-11-04 11:35:38,652 INFO scheduling health checks: 2019-11-04 11:35:38,706 INFO scheduling jobs: 2019-11-04 11:35:38,708 DEBUG - kvm 2019-11-04 11:35:38,785 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:35:38,785 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:35:38,790 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:35:40,041 INFO [INIT] Dropping privileges 2019-11-04 11:35:40,041 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:35:40,042 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:35:40,044 INFO [INIT] Marking all workers as offline 2019-11-04 11:35:40,073 INFO [INIT] Starting encryption 2019-11-04 11:35:40,075 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:35:40,075 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:35:40,076 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:35:40,087 INFO [INIT] LAVA master has started. 2019-11-04 11:35:40,087 INFO [INIT] Using protocol version 3 2019-11-04 11:35:40,233 DEBUG Dhanu => PING(20) 2019-11-04 11:35:40,234 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:35:42,700 DEBUG lava-logs => PING(20) 2019-11-04 11:36:00,100 INFO scheduling health checks: 2019-11-04 11:36:00,186 INFO scheduling jobs: 2019-11-04 11:36:00,187 DEBUG - kvm 2019-11-04 11:36:00,260 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:36:00,260 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:36:00,265 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:36:01,579 INFO [INIT] Dropping privileges 2019-11-04 11:36:01,579 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:36:01,579 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:36:01,581 INFO [INIT] Marking all workers as offline 2019-11-04 11:36:01,609 INFO [INIT] Starting encryption 2019-11-04 11:36:01,610 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:36:01,610 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:36:01,610 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:36:01,621 INFO [INIT] LAVA master has started. 2019-11-04 11:36:01,622 INFO [INIT] Using protocol version 3 2019-11-04 11:36:02,710 DEBUG lava-logs => PING(20) 2019-11-04 11:36:20,265 DEBUG Dhanu => PING(20) 2019-11-04 11:36:20,265 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:36:21,622 INFO scheduling health checks: 2019-11-04 11:36:21,666 INFO scheduling jobs: 2019-11-04 11:36:21,667 DEBUG - kvm 2019-11-04 11:36:21,755 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:36:21,755 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:36:21,761 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:36:23,007 INFO [INIT] Dropping privileges 2019-11-04 11:36:23,008 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:36:23,008 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:36:23,010 INFO [INIT] Marking all workers as offline 2019-11-04 11:36:23,066 INFO [INIT] Starting encryption 2019-11-04 11:36:23,067 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:36:23,067 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:36:23,068 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:36:23,078 INFO [INIT] LAVA master has started. 2019-11-04 11:36:23,079 INFO [INIT] Using protocol version 3 2019-11-04 11:36:23,220 DEBUG lava-logs => PING(20) 2019-11-04 11:36:40,277 DEBUG Dhanu => PING(20) 2019-11-04 11:36:40,277 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:36:43,080 INFO scheduling health checks: 2019-11-04 11:36:43,135 INFO scheduling jobs: 2019-11-04 11:36:43,137 DEBUG - kvm 2019-11-04 11:36:43,236 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:36:43,237 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:36:43,245 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:36:44,614 INFO [INIT] Dropping privileges 2019-11-04 11:36:44,614 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:36:44,615 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:36:44,617 INFO [INIT] Marking all workers as offline 2019-11-04 11:36:44,647 INFO [INIT] Starting encryption 2019-11-04 11:36:44,649 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:36:44,649 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:36:44,650 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:36:44,665 INFO [INIT] LAVA master has started. 2019-11-04 11:36:44,665 INFO [INIT] Using protocol version 3 2019-11-04 11:37:00,292 DEBUG Dhanu => PING(20) 2019-11-04 11:37:00,293 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:37:03,260 DEBUG lava-logs => PING(20) 2019-11-04 11:37:04,666 INFO scheduling health checks: 2019-11-04 11:37:04,715 INFO scheduling jobs: 2019-11-04 11:37:04,717 DEBUG - kvm 2019-11-04 11:37:04,792 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:37:04,793 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:37:04,797 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:37:05,978 INFO [INIT] Dropping privileges 2019-11-04 11:37:05,978 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:37:05,978 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:37:05,980 INFO [INIT] Marking all workers as offline 2019-11-04 11:37:06,005 INFO [INIT] Starting encryption 2019-11-04 11:37:06,007 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:37:06,007 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:37:06,007 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:37:06,023 INFO [INIT] LAVA master has started. 2019-11-04 11:37:06,023 INFO [INIT] Using protocol version 3 2019-11-04 11:37:20,314 DEBUG Dhanu => PING(20) 2019-11-04 11:37:20,314 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:37:23,278 DEBUG lava-logs => PING(20) 2019-11-04 11:37:26,027 INFO scheduling health checks: 2019-11-04 11:37:26,072 INFO scheduling jobs: 2019-11-04 11:37:26,074 DEBUG - kvm 2019-11-04 11:37:26,149 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:37:26,149 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:37:26,154 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:37:27,436 INFO [INIT] Dropping privileges 2019-11-04 11:37:27,436 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:37:27,437 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:37:27,438 INFO [INIT] Marking all workers as offline 2019-11-04 11:37:27,462 INFO [INIT] Starting encryption 2019-11-04 11:37:27,463 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:37:27,463 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:37:27,464 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:37:27,479 INFO [INIT] LAVA master has started. 2019-11-04 11:37:27,479 INFO [INIT] Using protocol version 3 2019-11-04 11:37:40,335 DEBUG Dhanu => PING(20) 2019-11-04 11:37:40,335 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:37:43,293 DEBUG lava-logs => PING(20) 2019-11-04 11:37:47,483 INFO scheduling health checks: 2019-11-04 11:37:47,541 INFO scheduling jobs: 2019-11-04 11:37:47,545 DEBUG - kvm 2019-11-04 11:37:47,622 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:37:47,622 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:37:47,627 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:37:49,104 INFO [INIT] Dropping privileges 2019-11-04 11:37:49,105 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:37:49,105 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:37:49,106 INFO [INIT] Marking all workers as offline 2019-11-04 11:37:49,131 INFO [INIT] Starting encryption 2019-11-04 11:37:49,133 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:37:49,134 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:37:49,134 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:37:49,151 INFO [INIT] LAVA master has started. 2019-11-04 11:37:49,151 INFO [INIT] Using protocol version 3 2019-11-04 11:38:00,414 DEBUG Dhanu => PING(20) 2019-11-04 11:38:00,425 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:38:03,364 DEBUG lava-logs => PING(20) 2019-11-04 11:38:09,157 INFO scheduling health checks: 2019-11-04 11:38:09,343 INFO scheduling jobs: 2019-11-04 11:38:09,344 DEBUG - kvm 2019-11-04 11:38:09,449 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:38:09,449 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:38:09,689 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:38:24,094 INFO [INIT] Dropping privileges 2019-11-04 11:38:24,119 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:38:24,119 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:38:24,123 INFO [INIT] Marking all workers as offline 2019-11-04 11:38:24,145 INFO [INIT] Starting encryption 2019-11-04 11:38:24,146 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:38:24,164 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:38:24,165 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:38:24,209 INFO [INIT] LAVA master has started. 2019-11-04 11:38:24,209 INFO [INIT] Using protocol version 3 2019-11-04 11:38:24,379 DEBUG Dhanu => PING(20) 2019-11-04 11:38:24,379 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:38:24,412 DEBUG lava-logs => PING(20) 2019-11-04 11:38:44,217 INFO scheduling health checks: 2019-11-04 11:38:44,514 INFO scheduling jobs: 2019-11-04 11:38:44,517 DEBUG - kvm 2019-11-04 11:38:44,674 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:38:44,674 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:38:44,759 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:38:48,954 INFO [INIT] Dropping privileges 2019-11-04 11:38:48,954 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:38:48,954 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:38:48,956 INFO [INIT] Marking all workers as offline 2019-11-04 11:38:48,992 INFO [INIT] Starting encryption 2019-11-04 11:38:48,994 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:38:48,994 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:38:48,994 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:38:49,008 INFO [INIT] LAVA master has started. 2019-11-04 11:38:49,009 INFO [INIT] Using protocol version 3 2019-11-04 11:39:04,446 DEBUG Dhanu => PING(20) 2019-11-04 11:39:04,447 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:39:04,482 DEBUG lava-logs => PING(20) 2019-11-04 11:39:09,014 INFO scheduling health checks: 2019-11-04 11:39:09,071 INFO scheduling jobs: 2019-11-04 11:39:09,073 DEBUG - kvm 2019-11-04 11:39:09,141 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:39:09,141 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:39:09,151 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:39:10,410 INFO [INIT] Dropping privileges 2019-11-04 11:39:10,411 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:39:10,411 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:39:10,413 INFO [INIT] Marking all workers as offline 2019-11-04 11:39:10,439 INFO [INIT] Starting encryption 2019-11-04 11:39:10,441 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:39:10,441 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:39:10,442 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:39:10,455 INFO [INIT] LAVA master has started. 2019-11-04 11:39:10,455 INFO [INIT] Using protocol version 3 2019-11-04 11:39:24,470 DEBUG Dhanu => PING(20) 2019-11-04 11:39:24,470 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:39:24,517 DEBUG lava-logs => PING(20) 2019-11-04 11:39:30,462 INFO scheduling health checks: 2019-11-04 11:39:30,518 INFO scheduling jobs: 2019-11-04 11:39:30,521 DEBUG - kvm 2019-11-04 11:39:30,597 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:39:30,597 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:39:30,602 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:39:31,910 INFO [INIT] Dropping privileges 2019-11-04 11:39:31,910 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:39:31,911 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:39:31,912 INFO [INIT] Marking all workers as offline 2019-11-04 11:39:31,941 INFO [INIT] Starting encryption 2019-11-04 11:39:31,942 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:39:31,942 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:39:31,943 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:39:31,954 INFO [INIT] LAVA master has started. 2019-11-04 11:39:31,954 INFO [INIT] Using protocol version 3 2019-11-04 11:39:44,499 DEBUG Dhanu => PING(20) 2019-11-04 11:39:44,499 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:39:44,553 DEBUG lava-logs => PING(20) 2019-11-04 11:39:51,960 INFO scheduling health checks: 2019-11-04 11:39:52,020 INFO scheduling jobs: 2019-11-04 11:39:52,024 DEBUG - kvm 2019-11-04 11:39:52,112 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:39:52,112 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:39:52,117 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:39:55,569 INFO [INIT] Dropping privileges 2019-11-04 11:39:55,570 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:39:55,570 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:39:55,572 INFO [INIT] Marking all workers as offline 2019-11-04 11:39:55,633 INFO [INIT] Starting encryption 2019-11-04 11:39:55,634 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:39:55,635 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:39:55,635 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:39:55,652 INFO [INIT] LAVA master has started. 2019-11-04 11:39:55,652 INFO [INIT] Using protocol version 3 2019-11-04 11:40:04,537 DEBUG Dhanu => PING(20) 2019-11-04 11:40:04,538 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:40:04,577 DEBUG lava-logs => PING(20) 2019-11-04 11:40:15,660 INFO scheduling health checks: 2019-11-04 11:40:15,689 INFO scheduling jobs: 2019-11-04 11:40:15,690 DEBUG - kvm 2019-11-04 11:40:15,780 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:40:15,780 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:40:15,786 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:40:17,096 INFO [INIT] Dropping privileges 2019-11-04 11:40:17,096 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:40:17,096 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:40:17,098 INFO [INIT] Marking all workers as offline 2019-11-04 11:40:17,123 INFO [INIT] Starting encryption 2019-11-04 11:40:17,125 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:40:17,126 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:40:17,126 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:40:17,138 INFO [INIT] LAVA master has started. 2019-11-04 11:40:17,139 INFO [INIT] Using protocol version 3 2019-11-04 11:40:24,607 DEBUG Dhanu => PING(20) 2019-11-04 11:40:24,623 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:40:24,690 DEBUG lava-logs => PING(20) 2019-11-04 11:40:37,171 INFO scheduling health checks: 2019-11-04 11:40:37,235 INFO scheduling jobs: 2019-11-04 11:40:37,237 DEBUG - kvm 2019-11-04 11:40:37,311 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:40:37,311 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:40:37,321 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:40:47,039 INFO [INIT] Dropping privileges 2019-11-04 11:40:47,039 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:40:47,039 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:40:47,041 INFO [INIT] Marking all workers as offline 2019-11-04 11:40:47,070 INFO [INIT] Starting encryption 2019-11-04 11:40:47,072 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:40:47,072 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:40:47,072 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:40:47,088 INFO [INIT] LAVA master has started. 2019-11-04 11:40:47,088 INFO [INIT] Using protocol version 3 2019-11-04 11:40:47,090 DEBUG lava-logs => PING(20) 2019-11-04 11:40:47,176 DEBUG Dhanu => PING(20) 2019-11-04 11:40:47,176 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:41:07,118 INFO scheduling health checks: 2019-11-04 11:41:07,283 INFO scheduling jobs: 2019-11-04 11:41:07,287 DEBUG - kvm 2019-11-04 11:41:07,382 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:41:07,382 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:41:07,393 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:41:11,799 INFO [INIT] Dropping privileges 2019-11-04 11:41:11,800 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:41:11,800 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:41:11,802 INFO [INIT] Marking all workers as offline 2019-11-04 11:41:11,840 INFO [INIT] Starting encryption 2019-11-04 11:41:11,841 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:41:11,842 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:41:11,842 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:41:11,861 INFO [INIT] LAVA master has started. 2019-11-04 11:41:11,861 INFO [INIT] Using protocol version 3 2019-11-04 11:41:27,126 DEBUG lava-logs => PING(20) 2019-11-04 11:41:27,233 DEBUG Dhanu => PING(20) 2019-11-04 11:41:27,233 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:41:31,866 INFO scheduling health checks: 2019-11-04 11:41:31,919 INFO scheduling jobs: 2019-11-04 11:41:31,921 DEBUG - kvm 2019-11-04 11:41:31,996 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:41:31,997 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:41:32,007 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:41:35,826 INFO [INIT] Dropping privileges 2019-11-04 11:41:35,826 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:41:35,827 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:41:35,828 INFO [INIT] Marking all workers as offline 2019-11-04 11:41:35,864 INFO [INIT] Starting encryption 2019-11-04 11:41:35,866 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:41:35,866 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:41:35,867 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:41:35,881 INFO [INIT] LAVA master has started. 2019-11-04 11:41:35,881 INFO [INIT] Using protocol version 3 2019-11-04 11:41:47,140 DEBUG lava-logs => PING(20) 2019-11-04 11:41:47,246 DEBUG Dhanu => PING(20) 2019-11-04 11:41:47,246 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:41:55,890 INFO scheduling health checks: 2019-11-04 11:41:55,944 INFO scheduling jobs: 2019-11-04 11:41:55,945 DEBUG - kvm 2019-11-04 11:41:56,022 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:41:56,022 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:41:56,027 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:41:58,484 INFO [INIT] Dropping privileges 2019-11-04 11:41:58,485 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:41:58,485 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:41:58,487 INFO [INIT] Marking all workers as offline 2019-11-04 11:41:58,522 INFO [INIT] Starting encryption 2019-11-04 11:41:58,523 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:41:58,523 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:41:58,523 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:41:58,547 INFO [INIT] LAVA master has started. 2019-11-04 11:41:58,547 INFO [INIT] Using protocol version 3 2019-11-04 11:42:07,154 DEBUG lava-logs => PING(20) 2019-11-04 11:42:07,266 DEBUG Dhanu => PING(20) 2019-11-04 11:42:07,266 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:42:18,559 INFO scheduling health checks: 2019-11-04 11:42:18,612 INFO scheduling jobs: 2019-11-04 11:42:18,614 DEBUG - kvm 2019-11-04 11:42:18,708 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:42:18,708 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:42:18,713 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:42:20,964 INFO [INIT] Dropping privileges 2019-11-04 11:42:20,965 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:42:20,965 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:42:20,967 INFO [INIT] Marking all workers as offline 2019-11-04 11:42:21,036 INFO [INIT] Starting encryption 2019-11-04 11:42:21,038 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:42:21,038 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d 2019-11-04 11:42:21,039 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d 2019-11-04 11:42:21,053 INFO [INIT] LAVA master has started. 2019-11-04 11:42:21,053 INFO [INIT] Using protocol version 3 2019-11-04 11:42:27,172 DEBUG lava-logs => PING(20) 2019-11-04 11:42:27,284 DEBUG Dhanu => PING(20) 2019-11-04 11:42:27,284 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:42:41,067 INFO scheduling health checks: 2019-11-04 11:42:41,148 INFO scheduling jobs: 2019-11-04 11:42:41,150 DEBUG - kvm 2019-11-04 11:42:41,226 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:42:41,226 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:42:41,231 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:42:42,959 INFO [INIT] Dropping privileges 2019-11-04 11:42:42,959 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:42:42,959 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:42:42,961 INFO [INIT] Marking all workers as offline 2019-11-04 11:42:42,982 INFO [INIT] Starting encryption 2019-11-04 11:42:42,984 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:42,984 ERROR Invalid certificate file: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:44,181 INFO [INIT] Dropping privileges 2019-11-04 11:42:44,181 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:42:44,181 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:42:44,183 INFO [INIT] Marking all workers as offline 2019-11-04 11:42:44,216 INFO [INIT] Starting encryption 2019-11-04 11:42:44,217 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:44,218 ERROR Invalid certificate file: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:45,422 INFO [INIT] Dropping privileges 2019-11-04 11:42:45,423 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:42:45,423 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:42:45,424 INFO [INIT] Marking all workers as offline 2019-11-04 11:42:45,462 INFO [INIT] Starting encryption 2019-11-04 11:42:45,464 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:45,464 ERROR Invalid certificate file: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:46,691 INFO [INIT] Dropping privileges 2019-11-04 11:42:46,691 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:42:46,691 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:42:46,692 INFO [INIT] Marking all workers as offline 2019-11-04 11:42:46,751 INFO [INIT] Starting encryption 2019-11-04 11:42:46,752 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:42:46,752 ERROR Invalid certificate file: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:44:28,468 INFO [INIT] Dropping privileges 2019-11-04 11:44:28,469 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:44:28,469 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:44:28,470 INFO [INIT] Marking all workers as offline 2019-11-04 11:44:28,516 INFO [INIT] Starting encryption 2019-11-04 11:44:28,518 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:44:28,518 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:44:28,519 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:44:28,534 INFO [INIT] LAVA master has started. 2019-11-04 11:44:28,535 INFO [INIT] Using protocol version 3 2019-11-04 11:44:28,599 DEBUG lava-logs => PING(20) 2019-11-04 11:44:28,614 DEBUG lava-logs => PING(20) 2019-11-04 11:44:28,644 INFO Dhanu => HELLO 2019-11-04 11:44:28,644 WARNING New dispatcher 2019-11-04 11:44:28,670 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,671 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,726 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,726 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,737 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,737 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,748 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,748 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,759 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,759 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,770 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,770 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,781 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,781 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,792 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,793 WARNING Dispatcher was not confirmed 2019-11-04 11:44:28,803 INFO Dhanu => HELLO_RETRY 2019-11-04 11:44:28,803 WARNING Dispatcher was not confirmed 2019-11-04 11:44:33,810 DEBUG Dhanu => PING(20) 2019-11-04 11:44:41,917 DEBUG lava-logs => PING(20) 2019-11-04 11:44:41,938 DEBUG lava-logs => PING(20) 2019-11-04 11:44:48,541 INFO scheduling health checks: 2019-11-04 11:44:48,627 INFO scheduling jobs: 2019-11-04 11:44:48,631 DEBUG - kvm 2019-11-04 11:44:48,734 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:44:48,734 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:44:48,775 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:44:49,922 INFO [INIT] Dropping privileges 2019-11-04 11:44:49,923 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:44:49,923 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:44:49,925 INFO [INIT] Marking all workers as offline 2019-11-04 11:44:49,950 INFO [INIT] Starting encryption 2019-11-04 11:44:49,951 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:44:49,952 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:44:49,952 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:44:49,967 INFO [INIT] LAVA master has started. 2019-11-04 11:44:49,967 INFO [INIT] Using protocol version 3 2019-11-04 11:44:53,830 DEBUG Dhanu => PING(20) 2019-11-04 11:44:53,831 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:45:01,952 DEBUG lava-logs => PING(20) 2019-11-04 11:45:09,975 INFO scheduling health checks: 2019-11-04 11:45:10,029 INFO scheduling jobs: 2019-11-04 11:45:10,031 DEBUG - kvm 2019-11-04 11:45:10,107 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:45:10,107 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:45:10,115 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:45:11,410 INFO [INIT] Dropping privileges 2019-11-04 11:45:11,411 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:45:11,411 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:45:11,412 INFO [INIT] Marking all workers as offline 2019-11-04 11:45:11,441 INFO [INIT] Starting encryption 2019-11-04 11:45:11,442 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:45:11,442 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:45:11,443 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:45:11,454 INFO [INIT] LAVA master has started. 2019-11-04 11:45:11,455 INFO [INIT] Using protocol version 3 2019-11-04 11:45:13,849 DEBUG Dhanu => PING(20) 2019-11-04 11:45:13,849 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:45:21,966 DEBUG lava-logs => PING(20) 2019-11-04 11:45:31,460 INFO scheduling health checks: 2019-11-04 11:45:31,509 INFO scheduling jobs: 2019-11-04 11:45:31,512 DEBUG - kvm 2019-11-04 11:45:31,589 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:45:31,589 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:45:31,593 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:45:32,908 INFO [INIT] Dropping privileges 2019-11-04 11:45:32,908 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:45:32,908 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:45:32,910 INFO [INIT] Marking all workers as offline 2019-11-04 11:45:32,967 INFO [INIT] Starting encryption 2019-11-04 11:45:32,967 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:45:32,967 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:45:32,967 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:45:32,976 INFO [INIT] LAVA master has started. 2019-11-04 11:45:32,977 INFO [INIT] Using protocol version 3 2019-11-04 11:45:33,869 DEBUG Dhanu => PING(20) 2019-11-04 11:45:33,870 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:45:37,111 DEBUG lava-logs => PING(20) 2019-11-04 11:45:37,132 DEBUG lava-logs => PING(20) 2019-11-04 11:45:51,452 INFO [POLL] Received a signal, leaving 2019-11-04 11:45:51,453 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:45:52,428 INFO [INIT] Dropping privileges 2019-11-04 11:45:52,429 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:45:52,429 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:45:52,431 INFO [INIT] Marking all workers as offline 2019-11-04 11:45:52,456 INFO [INIT] Starting encryption 2019-11-04 11:45:52,457 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:45:52,457 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:45:52,458 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:45:52,471 INFO [INIT] LAVA master has started. 2019-11-04 11:45:52,472 INFO [INIT] Using protocol version 3 2019-11-04 11:45:53,888 DEBUG Dhanu => PING(20) 2019-11-04 11:45:53,889 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:45:57,150 DEBUG lava-logs => PING(20) 2019-11-04 11:45:57,721 INFO Dhanu => HELLO 2019-11-04 11:45:57,721 WARNING Dispatcher has RESTARTED 2019-11-04 11:46:02,728 DEBUG Dhanu => PING(20) 2019-11-04 11:46:12,480 INFO scheduling health checks: 2019-11-04 11:46:12,535 INFO scheduling jobs: 2019-11-04 11:46:12,538 DEBUG - kvm 2019-11-04 11:46:12,618 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:46:12,618 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:46:12,623 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:46:14,112 INFO [INIT] Dropping privileges 2019-11-04 11:46:14,112 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:46:14,113 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:46:14,114 INFO [INIT] Marking all workers as offline 2019-11-04 11:46:14,181 INFO [INIT] Starting encryption 2019-11-04 11:46:14,183 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:46:14,183 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:46:14,184 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:46:14,204 INFO [INIT] LAVA master has started. 2019-11-04 11:46:14,204 INFO [INIT] Using protocol version 3 2019-11-04 11:46:17,165 DEBUG lava-logs => PING(20) 2019-11-04 11:46:22,746 DEBUG Dhanu => PING(20) 2019-11-04 11:46:22,746 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:46:34,215 INFO scheduling health checks: 2019-11-04 11:46:34,271 INFO scheduling jobs: 2019-11-04 11:46:34,273 DEBUG - kvm 2019-11-04 11:46:34,364 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:46:34,364 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:46:34,372 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:46:35,855 INFO [INIT] Dropping privileges 2019-11-04 11:46:35,855 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:46:35,855 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:46:35,857 INFO [INIT] Marking all workers as offline 2019-11-04 11:46:35,885 INFO [INIT] Starting encryption 2019-11-04 11:46:35,886 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:46:35,887 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:46:35,887 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:46:35,904 INFO [INIT] LAVA master has started. 2019-11-04 11:46:35,904 INFO [INIT] Using protocol version 3 2019-11-04 11:46:37,175 DEBUG lava-logs => PING(20) 2019-11-04 11:46:42,761 DEBUG Dhanu => PING(20) 2019-11-04 11:46:42,762 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:46:55,917 INFO scheduling health checks: 2019-11-04 11:46:55,973 INFO scheduling jobs: 2019-11-04 11:46:55,976 DEBUG - kvm 2019-11-04 11:46:56,058 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:46:56,058 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:46:56,063 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:46:57,495 INFO [INIT] Dropping privileges 2019-11-04 11:46:57,496 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:46:57,496 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:46:57,498 INFO [INIT] Marking all workers as offline 2019-11-04 11:46:57,531 INFO [INIT] Starting encryption 2019-11-04 11:46:57,533 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:46:57,534 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:46:57,534 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:46:57,551 INFO [INIT] LAVA master has started. 2019-11-04 11:46:57,551 INFO [INIT] Using protocol version 3 2019-11-04 11:46:57,717 DEBUG lava-logs => PING(20) 2019-11-04 11:47:02,776 DEBUG Dhanu => PING(20) 2019-11-04 11:47:02,776 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:47:17,566 INFO scheduling health checks: 2019-11-04 11:47:17,620 INFO scheduling jobs: 2019-11-04 11:47:17,622 DEBUG - kvm 2019-11-04 11:47:17,696 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:47:17,696 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:47:17,704 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:47:19,047 INFO [INIT] Dropping privileges 2019-11-04 11:47:19,047 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:47:19,047 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:47:19,049 INFO [INIT] Marking all workers as offline 2019-11-04 11:47:19,077 INFO [INIT] Starting encryption 2019-11-04 11:47:19,078 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:47:19,079 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:47:19,079 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:47:19,091 INFO [INIT] LAVA master has started. 2019-11-04 11:47:19,092 INFO [INIT] Using protocol version 3 2019-11-04 11:47:19,200 DEBUG lava-logs => PING(20) 2019-11-04 11:47:22,793 DEBUG Dhanu => PING(20) 2019-11-04 11:47:22,794 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:47:39,093 INFO scheduling health checks: 2019-11-04 11:47:39,144 INFO scheduling jobs: 2019-11-04 11:47:39,146 DEBUG - kvm 2019-11-04 11:47:39,222 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:47:39,222 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:47:39,232 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:47:40,591 INFO [INIT] Dropping privileges 2019-11-04 11:47:40,591 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:47:40,592 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:47:40,593 INFO [INIT] Marking all workers as offline 2019-11-04 11:47:40,623 INFO [INIT] Starting encryption 2019-11-04 11:47:40,624 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:47:40,625 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:47:40,625 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:47:40,642 INFO [INIT] LAVA master has started. 2019-11-04 11:47:40,643 INFO [INIT] Using protocol version 3 2019-11-04 11:47:42,811 DEBUG Dhanu => PING(20) 2019-11-04 11:47:42,812 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:47:59,266 DEBUG lava-logs => PING(20) 2019-11-04 11:48:00,644 INFO scheduling health checks: 2019-11-04 11:48:00,735 INFO scheduling jobs: 2019-11-04 11:48:00,739 DEBUG - kvm 2019-11-04 11:48:00,844 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:48:00,845 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:48:00,893 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:48:04,239 INFO [INIT] Dropping privileges 2019-11-04 11:48:04,239 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:48:04,239 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:48:04,241 INFO [INIT] Marking all workers as offline 2019-11-04 11:48:04,281 INFO [INIT] Starting encryption 2019-11-04 11:48:04,283 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:48:04,283 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:48:04,283 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:48:04,295 INFO [INIT] LAVA master has started. 2019-11-04 11:48:04,295 INFO [INIT] Using protocol version 3 2019-11-04 11:48:04,384 DEBUG Dhanu => PING(20) 2019-11-04 11:48:04,385 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:48:19,284 DEBUG lava-logs => PING(20) 2019-11-04 11:48:24,300 INFO scheduling health checks: 2019-11-04 11:48:24,371 INFO scheduling jobs: 2019-11-04 11:48:24,372 DEBUG - kvm 2019-11-04 11:48:24,439 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:48:24,439 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:48:24,443 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:48:25,693 INFO [INIT] Dropping privileges 2019-11-04 11:48:25,694 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:48:25,694 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:48:25,696 INFO [INIT] Marking all workers as offline 2019-11-04 11:48:25,727 INFO [INIT] Starting encryption 2019-11-04 11:48:25,728 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:48:25,728 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:48:25,729 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:48:25,742 INFO [INIT] LAVA master has started. 2019-11-04 11:48:25,742 INFO [INIT] Using protocol version 3 2019-11-04 11:48:39,297 DEBUG lava-logs => PING(20) 2019-11-04 11:48:44,421 DEBUG Dhanu => PING(20) 2019-11-04 11:48:44,422 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:48:45,743 INFO scheduling health checks: 2019-11-04 11:48:45,773 INFO scheduling jobs: 2019-11-04 11:48:45,774 DEBUG - kvm 2019-11-04 11:48:45,870 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:48:45,872 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:48:45,881 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:48:47,335 INFO [INIT] Dropping privileges 2019-11-04 11:48:47,335 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:48:47,335 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:48:47,337 INFO [INIT] Marking all workers as offline 2019-11-04 11:48:47,373 INFO [INIT] Starting encryption 2019-11-04 11:48:47,374 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:48:47,375 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:48:47,375 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:48:47,386 INFO [INIT] LAVA master has started. 2019-11-04 11:48:47,386 INFO [INIT] Using protocol version 3 2019-11-04 11:48:59,309 DEBUG lava-logs => PING(20) 2019-11-04 11:49:04,439 DEBUG Dhanu => PING(20) 2019-11-04 11:49:04,439 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:49:07,389 INFO scheduling health checks: 2019-11-04 11:49:07,453 INFO scheduling jobs: 2019-11-04 11:49:07,454 DEBUG - kvm 2019-11-04 11:49:07,526 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:49:07,526 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:49:07,534 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:49:08,669 INFO [INIT] Dropping privileges 2019-11-04 11:49:08,669 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:49:08,670 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:49:08,671 INFO [INIT] Marking all workers as offline 2019-11-04 11:49:08,710 INFO [INIT] Starting encryption 2019-11-04 11:49:08,711 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:49:08,712 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:49:08,713 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:49:08,726 INFO [INIT] LAVA master has started. 2019-11-04 11:49:08,727 INFO [INIT] Using protocol version 3 2019-11-04 11:49:19,323 DEBUG lava-logs => PING(20) 2019-11-04 11:49:24,459 DEBUG Dhanu => PING(20) 2019-11-04 11:49:24,460 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:49:28,731 INFO scheduling health checks: 2019-11-04 11:49:28,788 INFO scheduling jobs: 2019-11-04 11:49:28,790 DEBUG - kvm 2019-11-04 11:49:28,865 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:49:28,865 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:49:28,870 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:49:30,234 INFO [INIT] Dropping privileges 2019-11-04 11:49:30,234 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:49:30,234 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:49:30,236 INFO [INIT] Marking all workers as offline 2019-11-04 11:49:30,277 INFO [INIT] Starting encryption 2019-11-04 11:49:30,278 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:49:30,279 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:49:30,279 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:49:30,291 INFO [INIT] LAVA master has started. 2019-11-04 11:49:30,291 INFO [INIT] Using protocol version 3 2019-11-04 11:49:39,346 DEBUG lava-logs => PING(20) 2019-11-04 11:49:44,480 DEBUG Dhanu => PING(20) 2019-11-04 11:49:44,481 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:49:50,297 INFO scheduling health checks: 2019-11-04 11:49:50,368 INFO scheduling jobs: 2019-11-04 11:49:50,370 DEBUG - kvm 2019-11-04 11:49:50,443 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:49:50,443 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:49:50,448 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:49:51,719 INFO [INIT] Dropping privileges 2019-11-04 11:49:51,719 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:49:51,719 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:49:51,721 INFO [INIT] Marking all workers as offline 2019-11-04 11:49:51,748 INFO [INIT] Starting encryption 2019-11-04 11:49:51,749 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:49:51,750 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:49:51,750 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:49:51,769 INFO [INIT] LAVA master has started. 2019-11-04 11:49:51,769 INFO [INIT] Using protocol version 3 2019-11-04 11:49:59,366 DEBUG lava-logs => PING(20) 2019-11-04 11:50:04,498 DEBUG Dhanu => PING(20) 2019-11-04 11:50:04,498 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:50:11,777 INFO scheduling health checks: 2019-11-04 11:50:11,826 INFO scheduling jobs: 2019-11-04 11:50:11,827 DEBUG - kvm 2019-11-04 11:50:11,900 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:50:11,901 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:50:11,909 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:50:13,197 INFO [INIT] Dropping privileges 2019-11-04 11:50:13,197 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:50:13,198 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:50:13,199 INFO [INIT] Marking all workers as offline 2019-11-04 11:50:13,238 INFO [INIT] Starting encryption 2019-11-04 11:50:13,240 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:50:13,240 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:50:13,241 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:50:13,254 INFO [INIT] LAVA master has started. 2019-11-04 11:50:13,254 INFO [INIT] Using protocol version 3 2019-11-04 11:50:19,379 DEBUG lava-logs => PING(20) 2019-11-04 11:50:24,518 DEBUG Dhanu => PING(20) 2019-11-04 11:50:24,519 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:50:33,260 INFO scheduling health checks: 2019-11-04 11:50:33,305 INFO scheduling jobs: 2019-11-04 11:50:33,307 DEBUG - kvm 2019-11-04 11:50:33,386 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:50:33,386 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:50:33,395 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:50:34,806 INFO [INIT] Dropping privileges 2019-11-04 11:50:34,806 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:50:34,807 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:50:34,808 INFO [INIT] Marking all workers as offline 2019-11-04 11:50:34,840 INFO [INIT] Starting encryption 2019-11-04 11:50:34,842 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:50:34,842 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:50:34,843 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:50:34,855 INFO [INIT] LAVA master has started. 2019-11-04 11:50:34,856 INFO [INIT] Using protocol version 3 2019-11-04 11:50:39,397 DEBUG lava-logs => PING(20) 2019-11-04 11:50:44,540 DEBUG Dhanu => PING(20) 2019-11-04 11:50:44,541 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:50:54,866 INFO scheduling health checks: 2019-11-04 11:50:54,919 INFO scheduling jobs: 2019-11-04 11:50:54,921 DEBUG - kvm 2019-11-04 11:50:54,995 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:50:54,996 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:50:55,006 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:50:56,156 INFO [INIT] Dropping privileges 2019-11-04 11:50:56,156 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:50:56,156 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:50:56,158 INFO [INIT] Marking all workers as offline 2019-11-04 11:50:56,186 INFO [INIT] Starting encryption 2019-11-04 11:50:56,187 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:50:56,187 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:50:56,188 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:50:56,201 INFO [INIT] LAVA master has started. 2019-11-04 11:50:56,202 INFO [INIT] Using protocol version 3 2019-11-04 11:50:59,413 DEBUG lava-logs => PING(20) 2019-11-04 11:51:04,562 DEBUG Dhanu => PING(20) 2019-11-04 11:51:04,563 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:51:10,939 DEBUG lava-logs => PING(20) 2019-11-04 11:51:10,954 DEBUG lava-logs => PING(20) 2019-11-04 11:51:13,059 INFO [POLL] Received a signal, leaving 2019-11-04 11:51:13,059 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:51:14,032 INFO [INIT] Dropping privileges 2019-11-04 11:51:14,032 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:51:14,033 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:51:14,034 INFO [INIT] Marking all workers as offline 2019-11-04 11:51:14,067 INFO [INIT] Starting encryption 2019-11-04 11:51:14,069 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:51:14,069 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:51:14,070 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:51:14,086 INFO [INIT] LAVA master has started. 2019-11-04 11:51:14,087 INFO [INIT] Using protocol version 3 2019-11-04 11:51:18,232 INFO Dhanu => HELLO 2019-11-04 11:51:18,232 WARNING New dispatcher 2019-11-04 11:51:23,237 DEBUG Dhanu => PING(20) 2019-11-04 11:51:30,973 DEBUG lava-logs => PING(20) 2019-11-04 11:51:34,090 INFO scheduling health checks: 2019-11-04 11:51:34,123 INFO scheduling jobs: 2019-11-04 11:51:34,124 DEBUG - kvm 2019-11-04 11:51:34,191 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:51:34,191 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:51:34,196 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:51:35,407 INFO [INIT] Dropping privileges 2019-11-04 11:51:35,407 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:51:35,407 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:51:35,409 INFO [INIT] Marking all workers as offline 2019-11-04 11:51:35,446 INFO [INIT] Starting encryption 2019-11-04 11:51:35,448 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:51:35,449 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:51:35,449 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:51:35,467 INFO [INIT] LAVA master has started. 2019-11-04 11:51:35,467 INFO [INIT] Using protocol version 3 2019-11-04 11:51:43,258 DEBUG Dhanu => PING(20) 2019-11-04 11:51:43,259 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:51:50,989 DEBUG lava-logs => PING(20) 2019-11-04 11:51:55,469 INFO scheduling health checks: 2019-11-04 11:51:55,514 INFO scheduling jobs: 2019-11-04 11:51:55,515 DEBUG - kvm 2019-11-04 11:51:55,589 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:51:55,590 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:51:55,595 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:51:57,180 INFO [INIT] Dropping privileges 2019-11-04 11:51:57,180 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:51:57,180 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:51:57,183 INFO [INIT] Marking all workers as offline 2019-11-04 11:51:57,214 INFO [INIT] Starting encryption 2019-11-04 11:51:57,215 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:51:57,215 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:51:57,215 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:51:57,229 INFO [INIT] LAVA master has started. 2019-11-04 11:51:57,229 INFO [INIT] Using protocol version 3 2019-11-04 11:52:03,334 DEBUG Dhanu => PING(20) 2019-11-04 11:52:03,370 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:52:11,032 DEBUG lava-logs => PING(20) 2019-11-04 11:52:17,233 INFO scheduling health checks: 2019-11-04 11:52:17,327 INFO scheduling jobs: 2019-11-04 11:52:17,331 DEBUG - kvm 2019-11-04 11:52:17,428 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:52:17,428 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:52:17,474 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:52:23,084 INFO [INIT] Dropping privileges 2019-11-04 11:52:23,084 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:52:23,084 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:52:23,086 INFO [INIT] Marking all workers as offline 2019-11-04 11:52:23,118 INFO [INIT] Starting encryption 2019-11-04 11:52:23,119 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:52:23,120 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:52:23,120 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:52:23,135 INFO [INIT] LAVA master has started. 2019-11-04 11:52:23,135 INFO [INIT] Using protocol version 3 2019-11-04 11:52:23,382 DEBUG Dhanu => PING(20) 2019-11-04 11:52:23,383 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:52:31,051 DEBUG lava-logs => PING(20) 2019-11-04 11:52:43,147 INFO scheduling health checks: 2019-11-04 11:52:43,207 INFO scheduling jobs: 2019-11-04 11:52:43,209 DEBUG - kvm 2019-11-04 11:52:43,280 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:52:43,280 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:52:43,285 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:52:44,433 INFO [INIT] Dropping privileges 2019-11-04 11:52:44,433 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:52:44,433 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:52:44,435 INFO [INIT] Marking all workers as offline 2019-11-04 11:52:44,464 INFO [INIT] Starting encryption 2019-11-04 11:52:44,465 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:52:44,466 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:52:44,466 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:52:44,478 INFO [INIT] LAVA master has started. 2019-11-04 11:52:44,479 INFO [INIT] Using protocol version 3 2019-11-04 11:52:44,537 DEBUG Dhanu => PING(20) 2019-11-04 11:52:44,538 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:52:51,065 DEBUG lava-logs => PING(20) 2019-11-04 11:53:04,489 INFO scheduling health checks: 2019-11-04 11:53:04,543 INFO scheduling jobs: 2019-11-04 11:53:04,545 DEBUG - kvm 2019-11-04 11:53:04,627 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:53:04,627 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:53:04,632 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:53:05,930 INFO [INIT] Dropping privileges 2019-11-04 11:53:05,930 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:53:05,930 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:53:05,932 INFO [INIT] Marking all workers as offline 2019-11-04 11:53:05,988 INFO [INIT] Starting encryption 2019-11-04 11:53:05,990 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:53:05,991 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:53:05,991 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:53:06,004 INFO [INIT] LAVA master has started. 2019-11-04 11:53:06,004 INFO [INIT] Using protocol version 3 2019-11-04 11:53:11,084 DEBUG lava-logs => PING(20) 2019-11-04 11:53:24,728 DEBUG Dhanu => PING(20) 2019-11-04 11:53:24,733 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:53:26,006 INFO scheduling health checks: 2019-11-04 11:53:26,067 INFO scheduling jobs: 2019-11-04 11:53:26,071 DEBUG - kvm 2019-11-04 11:53:26,202 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:53:26,203 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:53:26,248 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:53:33,687 INFO [INIT] Dropping privileges 2019-11-04 11:53:33,688 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:53:33,688 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:53:33,691 INFO [INIT] Marking all workers as offline 2019-11-04 11:53:33,724 INFO [INIT] Starting encryption 2019-11-04 11:53:33,725 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:53:33,725 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:53:33,725 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:53:33,735 INFO [INIT] LAVA master has started. 2019-11-04 11:53:33,735 INFO [INIT] Using protocol version 3 2019-11-04 11:53:33,758 DEBUG lava-logs => PING(20) 2019-11-04 11:53:44,747 DEBUG Dhanu => PING(20) 2019-11-04 11:53:44,748 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:53:53,742 INFO scheduling health checks: 2019-11-04 11:53:53,792 INFO scheduling jobs: 2019-11-04 11:53:53,794 DEBUG - kvm 2019-11-04 11:53:53,866 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:53:53,866 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:53:53,871 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:53:55,202 INFO [INIT] Dropping privileges 2019-11-04 11:53:55,203 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:53:55,203 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:53:55,205 INFO [INIT] Marking all workers as offline 2019-11-04 11:53:55,248 INFO [INIT] Starting encryption 2019-11-04 11:53:55,250 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:53:55,251 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:53:55,251 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:53:55,265 INFO [INIT] LAVA master has started. 2019-11-04 11:53:55,265 INFO [INIT] Using protocol version 3 2019-11-04 11:54:04,765 DEBUG Dhanu => PING(20) 2019-11-04 11:54:04,766 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:54:13,796 DEBUG lava-logs => PING(20) 2019-11-04 11:54:15,267 INFO scheduling health checks: 2019-11-04 11:54:15,316 INFO scheduling jobs: 2019-11-04 11:54:15,318 DEBUG - kvm 2019-11-04 11:54:15,391 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:54:15,391 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:54:15,396 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:54:16,651 INFO [INIT] Dropping privileges 2019-11-04 11:54:16,651 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:54:16,651 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:54:16,653 INFO [INIT] Marking all workers as offline 2019-11-04 11:54:16,684 INFO [INIT] Starting encryption 2019-11-04 11:54:16,686 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:54:16,686 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:54:16,687 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:54:16,699 INFO [INIT] LAVA master has started. 2019-11-04 11:54:16,700 INFO [INIT] Using protocol version 3 2019-11-04 11:54:24,785 DEBUG Dhanu => PING(20) 2019-11-04 11:54:24,785 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:54:33,813 DEBUG lava-logs => PING(20) 2019-11-04 11:54:36,702 INFO scheduling health checks: 2019-11-04 11:54:36,807 INFO scheduling jobs: 2019-11-04 11:54:36,809 DEBUG - kvm 2019-11-04 11:54:36,978 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:54:36,978 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:54:37,030 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:54:51,126 INFO [INIT] Dropping privileges 2019-11-04 11:54:51,160 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:54:51,160 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:54:51,183 INFO [INIT] Marking all workers as offline 2019-11-04 11:54:51,267 INFO [INIT] Starting encryption 2019-11-04 11:54:51,268 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:54:51,317 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:54:51,317 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:54:51,438 INFO [INIT] LAVA master has started. 2019-11-04 11:54:51,439 INFO [INIT] Using protocol version 3 2019-11-04 11:54:51,596 DEBUG Dhanu => PING(20) 2019-11-04 11:54:51,597 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:54:54,169 DEBUG lava-logs => PING(20) 2019-11-04 11:55:11,456 INFO scheduling health checks: 2019-11-04 11:55:11,588 INFO scheduling jobs: 2019-11-04 11:55:11,590 DEBUG - kvm 2019-11-04 11:55:11,702 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:55:11,702 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:55:11,764 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:55:21,097 INFO [INIT] Dropping privileges 2019-11-04 11:55:21,097 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:55:21,097 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:55:21,099 INFO [INIT] Marking all workers as offline 2019-11-04 11:55:21,134 INFO [INIT] Starting encryption 2019-11-04 11:55:21,135 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:55:21,135 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:55:21,135 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:55:21,143 INFO [INIT] LAVA master has started. 2019-11-04 11:55:21,144 INFO [INIT] Using protocol version 3 2019-11-04 11:55:21,146 DEBUG lava-logs => PING(20) 2019-11-04 11:55:31,640 DEBUG Dhanu => PING(20) 2019-11-04 11:55:31,641 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:55:41,153 INFO scheduling health checks: 2019-11-04 11:55:41,191 INFO scheduling jobs: 2019-11-04 11:55:41,193 DEBUG - kvm 2019-11-04 11:55:41,275 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:55:41,275 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:55:41,280 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:55:43,183 INFO [INIT] Dropping privileges 2019-11-04 11:55:43,183 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:55:43,183 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:55:43,185 INFO [INIT] Marking all workers as offline 2019-11-04 11:55:43,214 INFO [INIT] Starting encryption 2019-11-04 11:55:43,234 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:55:43,235 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:55:43,235 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:55:43,244 INFO [INIT] LAVA master has started. 2019-11-04 11:55:43,244 INFO [INIT] Using protocol version 3 2019-11-04 11:55:51,657 DEBUG Dhanu => PING(20) 2019-11-04 11:55:51,657 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:56:01,181 DEBUG lava-logs => PING(20) 2019-11-04 11:56:03,245 INFO scheduling health checks: 2019-11-04 11:56:03,316 INFO scheduling jobs: 2019-11-04 11:56:03,317 DEBUG - kvm 2019-11-04 11:56:03,391 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:56:03,392 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:56:03,397 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:56:05,227 INFO [INIT] Dropping privileges 2019-11-04 11:56:05,227 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:56:05,227 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:56:05,230 INFO [INIT] Marking all workers as offline 2019-11-04 11:56:05,284 INFO [INIT] Starting encryption 2019-11-04 11:56:05,285 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:56:05,286 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:56:05,286 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:56:05,298 INFO [INIT] LAVA master has started. 2019-11-04 11:56:05,299 INFO [INIT] Using protocol version 3 2019-11-04 11:56:11,671 DEBUG Dhanu => PING(20) 2019-11-04 11:56:11,671 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:56:21,193 DEBUG lava-logs => PING(20) 2019-11-04 11:56:25,300 INFO scheduling health checks: 2019-11-04 11:56:25,351 INFO scheduling jobs: 2019-11-04 11:56:25,353 DEBUG - kvm 2019-11-04 11:56:25,436 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:56:25,436 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:56:25,442 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:56:28,298 INFO [INIT] Dropping privileges 2019-11-04 11:56:28,298 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:56:28,298 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:56:28,301 INFO [INIT] Marking all workers as offline 2019-11-04 11:56:28,341 INFO [INIT] Starting encryption 2019-11-04 11:56:28,342 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:56:28,342 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:56:28,342 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:56:28,352 INFO [INIT] LAVA master has started. 2019-11-04 11:56:28,352 INFO [INIT] Using protocol version 3 2019-11-04 11:56:31,689 DEBUG Dhanu => PING(20) 2019-11-04 11:56:31,689 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:56:41,205 DEBUG lava-logs => PING(20) 2019-11-04 11:56:48,356 INFO scheduling health checks: 2019-11-04 11:56:48,409 INFO scheduling jobs: 2019-11-04 11:56:48,411 DEBUG - kvm 2019-11-04 11:56:48,507 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:56:48,507 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:56:48,515 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:56:50,256 INFO [INIT] Dropping privileges 2019-11-04 11:56:50,256 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:56:50,257 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:56:50,258 INFO [INIT] Marking all workers as offline 2019-11-04 11:56:50,288 INFO [INIT] Starting encryption 2019-11-04 11:56:50,289 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:56:50,290 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:56:50,290 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:56:50,303 INFO [INIT] LAVA master has started. 2019-11-04 11:56:50,303 INFO [INIT] Using protocol version 3 2019-11-04 11:56:51,704 DEBUG Dhanu => PING(20) 2019-11-04 11:56:51,704 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:57:01,217 DEBUG lava-logs => PING(20) 2019-11-04 11:57:10,312 INFO scheduling health checks: 2019-11-04 11:57:10,356 INFO scheduling jobs: 2019-11-04 11:57:10,357 DEBUG - kvm 2019-11-04 11:57:10,456 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:57:10,456 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:57:10,461 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:57:11,999 INFO [INIT] Dropping privileges 2019-11-04 11:57:12,000 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:57:12,000 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:57:12,002 INFO [INIT] Marking all workers as offline 2019-11-04 11:57:12,034 INFO [INIT] Starting encryption 2019-11-04 11:57:12,035 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:57:12,035 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:57:12,035 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:57:12,044 INFO [INIT] LAVA master has started. 2019-11-04 11:57:12,044 INFO [INIT] Using protocol version 3 2019-11-04 11:57:12,113 DEBUG Dhanu => PING(20) 2019-11-04 11:57:12,114 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:57:21,232 DEBUG lava-logs => PING(20) 2019-11-04 11:57:32,055 INFO scheduling health checks: 2019-11-04 11:57:32,136 INFO scheduling jobs: 2019-11-04 11:57:32,137 DEBUG - kvm 2019-11-04 11:57:32,207 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:57:32,208 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:57:32,215 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:57:33,516 INFO [INIT] Dropping privileges 2019-11-04 11:57:33,516 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:57:33,516 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:57:33,518 INFO [INIT] Marking all workers as offline 2019-11-04 11:57:33,559 INFO [INIT] Starting encryption 2019-11-04 11:57:33,560 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:57:33,560 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:57:33,561 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:57:33,574 INFO [INIT] LAVA master has started. 2019-11-04 11:57:33,574 INFO [INIT] Using protocol version 3 2019-11-04 11:57:41,249 DEBUG lava-logs => PING(20) 2019-11-04 11:57:52,145 DEBUG Dhanu => PING(20) 2019-11-04 11:57:52,145 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:57:53,575 INFO scheduling health checks: 2019-11-04 11:57:53,616 INFO scheduling jobs: 2019-11-04 11:57:53,617 DEBUG - kvm 2019-11-04 11:57:53,691 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:57:53,691 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:57:53,704 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:57:55,532 INFO [INIT] Dropping privileges 2019-11-04 11:57:55,532 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:57:55,536 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:57:55,539 INFO [INIT] Marking all workers as offline 2019-11-04 11:57:55,561 INFO [INIT] Starting encryption 2019-11-04 11:57:55,562 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:57:55,562 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:57:55,562 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:57:55,580 INFO [INIT] LAVA master has started. 2019-11-04 11:57:55,580 INFO [INIT] Using protocol version 3 2019-11-04 11:58:01,261 DEBUG lava-logs => PING(20) 2019-11-04 11:58:12,161 DEBUG Dhanu => PING(20) 2019-11-04 11:58:12,162 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:58:15,584 INFO scheduling health checks: 2019-11-04 11:58:15,662 INFO scheduling jobs: 2019-11-04 11:58:15,664 DEBUG - kvm 2019-11-04 11:58:15,734 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:58:15,734 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:58:15,745 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:58:16,969 INFO [INIT] Dropping privileges 2019-11-04 11:58:16,970 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:58:16,970 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:58:16,972 INFO [INIT] Marking all workers as offline 2019-11-04 11:58:16,997 INFO [INIT] Starting encryption 2019-11-04 11:58:16,999 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:58:17,000 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:58:17,000 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:58:17,014 INFO [INIT] LAVA master has started. 2019-11-04 11:58:17,014 INFO [INIT] Using protocol version 3 2019-11-04 11:58:21,278 DEBUG lava-logs => PING(20) 2019-11-04 11:58:32,178 DEBUG Dhanu => PING(20) 2019-11-04 11:58:32,178 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:58:37,019 INFO scheduling health checks: 2019-11-04 11:58:37,076 INFO scheduling jobs: 2019-11-04 11:58:37,078 DEBUG - kvm 2019-11-04 11:58:37,149 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:58:37,149 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:58:37,158 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:58:38,403 INFO [INIT] Dropping privileges 2019-11-04 11:58:38,403 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:58:38,404 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:58:38,405 INFO [INIT] Marking all workers as offline 2019-11-04 11:58:38,443 INFO [INIT] Starting encryption 2019-11-04 11:58:38,445 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:58:38,445 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:58:38,445 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:58:38,458 INFO [INIT] LAVA master has started. 2019-11-04 11:58:38,458 INFO [INIT] Using protocol version 3 2019-11-04 11:58:41,294 DEBUG lava-logs => PING(20) 2019-11-04 11:58:52,199 DEBUG Dhanu => PING(20) 2019-11-04 11:58:52,200 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:58:58,465 INFO scheduling health checks: 2019-11-04 11:58:58,511 INFO scheduling jobs: 2019-11-04 11:58:58,513 DEBUG - kvm 2019-11-04 11:58:58,583 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:58:58,583 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:58:58,588 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:58:59,657 INFO [INIT] Dropping privileges 2019-11-04 11:58:59,657 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:58:59,657 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:58:59,659 INFO [INIT] Marking all workers as offline 2019-11-04 11:58:59,690 INFO [INIT] Starting encryption 2019-11-04 11:58:59,692 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:58:59,692 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:58:59,693 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:58:59,709 INFO [INIT] LAVA master has started. 2019-11-04 11:58:59,709 INFO [INIT] Using protocol version 3 2019-11-04 11:59:01,308 DEBUG lava-logs => PING(20) 2019-11-04 11:59:12,221 DEBUG Dhanu => PING(20) 2019-11-04 11:59:12,221 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:59:19,716 INFO scheduling health checks: 2019-11-04 11:59:19,769 INFO scheduling jobs: 2019-11-04 11:59:19,771 DEBUG - kvm 2019-11-04 11:59:19,843 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:59:19,844 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:59:19,848 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:59:21,162 INFO [INIT] Dropping privileges 2019-11-04 11:59:21,162 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:59:21,163 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:59:21,164 INFO [INIT] Marking all workers as offline 2019-11-04 11:59:21,214 INFO [INIT] Starting encryption 2019-11-04 11:59:21,216 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:59:21,217 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:59:21,217 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:59:21,228 INFO [INIT] LAVA master has started. 2019-11-04 11:59:21,228 INFO [INIT] Using protocol version 3 2019-11-04 11:59:21,367 DEBUG lava-logs => PING(20) 2019-11-04 11:59:32,238 DEBUG Dhanu => PING(20) 2019-11-04 11:59:32,239 WARNING Unknown dispatcher (server crashed) 2019-11-04 11:59:41,230 INFO scheduling health checks: 2019-11-04 11:59:41,282 INFO scheduling jobs: 2019-11-04 11:59:41,284 DEBUG - kvm 2019-11-04 11:59:41,357 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 11:59:41,357 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 11:59:41,364 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 11:59:42,651 INFO [INIT] Dropping privileges 2019-11-04 11:59:42,651 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 11:59:42,651 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 11:59:42,653 INFO [INIT] Marking all workers as offline 2019-11-04 11:59:42,717 INFO [INIT] Starting encryption 2019-11-04 11:59:42,718 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 11:59:42,719 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:59:42,719 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 11:59:42,730 INFO [INIT] LAVA master has started. 2019-11-04 11:59:42,730 INFO [INIT] Using protocol version 3 2019-11-04 11:59:42,800 DEBUG lava-logs => PING(20) 2019-11-04 11:59:52,262 DEBUG Dhanu => PING(20) 2019-11-04 11:59:52,262 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:00:02,741 INFO scheduling health checks: 2019-11-04 12:00:02,795 INFO scheduling jobs: 2019-11-04 12:00:02,797 DEBUG - kvm 2019-11-04 12:00:02,869 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:00:02,869 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:00:02,874 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:00:04,166 INFO [INIT] Dropping privileges 2019-11-04 12:00:04,167 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:00:04,167 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:00:04,169 INFO [INIT] Marking all workers as offline 2019-11-04 12:00:04,196 INFO [INIT] Starting encryption 2019-11-04 12:00:04,198 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:00:04,198 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:00:04,199 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:00:04,211 INFO [INIT] LAVA master has started. 2019-11-04 12:00:04,211 INFO [INIT] Using protocol version 3 2019-11-04 12:00:12,284 DEBUG Dhanu => PING(20) 2019-11-04 12:00:12,285 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:00:22,843 DEBUG lava-logs => PING(20) 2019-11-04 12:00:24,213 INFO scheduling health checks: 2019-11-04 12:00:24,242 INFO scheduling jobs: 2019-11-04 12:00:24,243 DEBUG - kvm 2019-11-04 12:00:24,309 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:00:24,309 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:00:24,318 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:00:25,410 INFO [INIT] Dropping privileges 2019-11-04 12:00:25,410 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:00:25,411 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:00:25,412 INFO [INIT] Marking all workers as offline 2019-11-04 12:00:25,443 INFO [INIT] Starting encryption 2019-11-04 12:00:25,445 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:00:25,445 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:00:25,446 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:00:25,460 INFO [INIT] LAVA master has started. 2019-11-04 12:00:25,460 INFO [INIT] Using protocol version 3 2019-11-04 12:00:32,302 DEBUG Dhanu => PING(20) 2019-11-04 12:00:32,302 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:00:42,861 DEBUG lava-logs => PING(20) 2019-11-04 12:00:45,463 INFO scheduling health checks: 2019-11-04 12:00:45,522 INFO scheduling jobs: 2019-11-04 12:00:45,523 DEBUG - kvm 2019-11-04 12:00:45,590 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:00:45,591 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:00:45,598 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:00:46,904 INFO [INIT] Dropping privileges 2019-11-04 12:00:46,904 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:00:46,905 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:00:46,906 INFO [INIT] Marking all workers as offline 2019-11-04 12:00:46,934 INFO [INIT] Starting encryption 2019-11-04 12:00:46,936 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:00:46,936 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:00:46,937 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:00:46,950 INFO [INIT] LAVA master has started. 2019-11-04 12:00:46,950 INFO [INIT] Using protocol version 3 2019-11-04 12:00:52,318 DEBUG Dhanu => PING(20) 2019-11-04 12:00:52,318 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:01:02,880 DEBUG lava-logs => PING(20) 2019-11-04 12:01:06,952 INFO scheduling health checks: 2019-11-04 12:01:07,001 INFO scheduling jobs: 2019-11-04 12:01:07,003 DEBUG - kvm 2019-11-04 12:01:07,076 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:01:07,076 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:01:07,081 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:01:08,404 INFO [INIT] Dropping privileges 2019-11-04 12:01:08,404 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:01:08,404 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:01:08,406 INFO [INIT] Marking all workers as offline 2019-11-04 12:01:08,436 INFO [INIT] Starting encryption 2019-11-04 12:01:08,437 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:01:08,438 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:01:08,438 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:01:08,452 INFO [INIT] LAVA master has started. 2019-11-04 12:01:08,453 INFO [INIT] Using protocol version 3 2019-11-04 12:01:12,334 DEBUG Dhanu => PING(20) 2019-11-04 12:01:12,334 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:01:22,893 DEBUG lava-logs => PING(20) 2019-11-04 12:01:28,456 INFO scheduling health checks: 2019-11-04 12:01:28,494 INFO scheduling jobs: 2019-11-04 12:01:28,496 DEBUG - kvm 2019-11-04 12:01:28,575 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:01:28,575 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:01:28,580 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:01:30,020 INFO [INIT] Dropping privileges 2019-11-04 12:01:30,021 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:01:30,021 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:01:30,023 INFO [INIT] Marking all workers as offline 2019-11-04 12:01:30,050 INFO [INIT] Starting encryption 2019-11-04 12:01:30,052 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:01:30,052 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:01:30,053 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:01:30,067 INFO [INIT] LAVA master has started. 2019-11-04 12:01:30,067 INFO [INIT] Using protocol version 3 2019-11-04 12:01:32,352 DEBUG Dhanu => PING(20) 2019-11-04 12:01:32,353 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:01:42,909 DEBUG lava-logs => PING(20) 2019-11-04 12:01:50,075 INFO scheduling health checks: 2019-11-04 12:01:50,128 INFO scheduling jobs: 2019-11-04 12:01:50,131 DEBUG - kvm 2019-11-04 12:01:50,205 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:01:50,205 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:01:50,210 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:01:51,437 INFO [INIT] Dropping privileges 2019-11-04 12:01:51,438 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:01:51,438 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:01:51,439 INFO [INIT] Marking all workers as offline 2019-11-04 12:01:51,474 INFO [INIT] Starting encryption 2019-11-04 12:01:51,476 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:01:51,476 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:01:51,477 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:01:51,496 INFO [INIT] LAVA master has started. 2019-11-04 12:01:51,496 INFO [INIT] Using protocol version 3 2019-11-04 12:01:52,371 DEBUG Dhanu => PING(20) 2019-11-04 12:01:52,372 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:02:02,922 DEBUG lava-logs => PING(20) 2019-11-04 12:02:11,505 INFO scheduling health checks: 2019-11-04 12:02:11,553 INFO scheduling jobs: 2019-11-04 12:02:11,555 DEBUG - kvm 2019-11-04 12:02:11,628 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:02:11,629 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:02:11,633 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:02:12,915 INFO [INIT] Dropping privileges 2019-11-04 12:02:12,915 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:02:12,916 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:02:12,917 INFO [INIT] Marking all workers as offline 2019-11-04 12:02:12,942 INFO [INIT] Starting encryption 2019-11-04 12:02:12,943 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:02:12,944 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:02:12,944 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:02:12,960 INFO [INIT] LAVA master has started. 2019-11-04 12:02:12,960 INFO [INIT] Using protocol version 3 2019-11-04 12:02:13,104 DEBUG Dhanu => PING(20) 2019-11-04 12:02:13,105 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:02:22,945 DEBUG lava-logs => PING(20) 2019-11-04 12:02:32,970 INFO scheduling health checks: 2019-11-04 12:02:33,021 INFO scheduling jobs: 2019-11-04 12:02:33,022 DEBUG - kvm 2019-11-04 12:02:33,094 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:02:33,094 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:02:33,104 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:02:34,452 INFO [INIT] Dropping privileges 2019-11-04 12:02:34,453 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:02:34,453 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:02:34,455 INFO [INIT] Marking all workers as offline 2019-11-04 12:02:34,501 INFO [INIT] Starting encryption 2019-11-04 12:02:34,502 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:02:34,503 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:02:34,503 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:02:34,520 INFO [INIT] LAVA master has started. 2019-11-04 12:02:34,520 INFO [INIT] Using protocol version 3 2019-11-04 12:02:34,616 DEBUG Dhanu => PING(20) 2019-11-04 12:02:34,616 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:02:42,966 DEBUG lava-logs => PING(20) 2019-11-04 12:02:54,532 INFO scheduling health checks: 2019-11-04 12:02:54,590 INFO scheduling jobs: 2019-11-04 12:02:54,592 DEBUG - kvm 2019-11-04 12:02:54,665 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:02:54,665 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:02:54,673 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:02:55,940 INFO [INIT] Dropping privileges 2019-11-04 12:02:55,941 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:02:55,941 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:02:55,943 INFO [INIT] Marking all workers as offline 2019-11-04 12:02:55,980 INFO [INIT] Starting encryption 2019-11-04 12:02:55,981 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:02:55,981 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:02:55,982 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:02:55,998 INFO [INIT] LAVA master has started. 2019-11-04 12:02:55,999 INFO [INIT] Using protocol version 3 2019-11-04 12:03:02,985 DEBUG lava-logs => PING(20) 2019-11-04 12:03:14,658 DEBUG Dhanu => PING(20) 2019-11-04 12:03:14,658 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:03:16,000 INFO scheduling health checks: 2019-11-04 12:03:16,048 INFO scheduling jobs: 2019-11-04 12:03:16,050 DEBUG - kvm 2019-11-04 12:03:16,122 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:03:16,123 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:03:16,127 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:03:17,422 INFO [INIT] Dropping privileges 2019-11-04 12:03:17,422 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:03:17,422 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:03:17,424 INFO [INIT] Marking all workers as offline 2019-11-04 12:03:17,459 INFO [INIT] Starting encryption 2019-11-04 12:03:17,462 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:03:17,462 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:03:17,463 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:03:17,474 INFO [INIT] LAVA master has started. 2019-11-04 12:03:17,475 INFO [INIT] Using protocol version 3 2019-11-04 12:03:23,003 DEBUG lava-logs => PING(20) 2019-11-04 12:03:34,673 DEBUG Dhanu => PING(20) 2019-11-04 12:03:34,674 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:03:37,478 INFO scheduling health checks: 2019-11-04 12:03:37,539 INFO scheduling jobs: 2019-11-04 12:03:37,541 DEBUG - kvm 2019-11-04 12:03:37,612 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:03:37,613 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:03:37,620 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:03:38,931 INFO [INIT] Dropping privileges 2019-11-04 12:03:38,931 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:03:38,931 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:03:38,933 INFO [INIT] Marking all workers as offline 2019-11-04 12:03:38,962 INFO [INIT] Starting encryption 2019-11-04 12:03:38,964 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:03:38,965 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:03:38,965 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:03:38,979 INFO [INIT] LAVA master has started. 2019-11-04 12:03:38,979 INFO [INIT] Using protocol version 3 2019-11-04 12:03:43,014 DEBUG lava-logs => PING(20) 2019-11-04 12:03:54,693 DEBUG Dhanu => PING(20) 2019-11-04 12:03:54,693 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:03:58,984 INFO scheduling health checks: 2019-11-04 12:03:59,052 INFO scheduling jobs: 2019-11-04 12:03:59,053 DEBUG - kvm 2019-11-04 12:03:59,125 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:03:59,126 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:03:59,130 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:04:00,411 INFO [INIT] Dropping privileges 2019-11-04 12:04:00,412 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:04:00,412 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:04:00,413 INFO [INIT] Marking all workers as offline 2019-11-04 12:04:00,442 INFO [INIT] Starting encryption 2019-11-04 12:04:00,444 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:04:00,444 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:04:00,445 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:04:00,458 INFO [INIT] LAVA master has started. 2019-11-04 12:04:00,458 INFO [INIT] Using protocol version 3 2019-11-04 12:04:03,030 DEBUG lava-logs => PING(20) 2019-11-04 12:04:14,717 DEBUG Dhanu => PING(20) 2019-11-04 12:04:14,717 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:04:20,460 INFO scheduling health checks: 2019-11-04 12:04:20,521 INFO scheduling jobs: 2019-11-04 12:04:20,522 DEBUG - kvm 2019-11-04 12:04:20,586 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:04:20,587 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:04:20,591 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:04:21,915 INFO [INIT] Dropping privileges 2019-11-04 12:04:21,915 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:04:21,915 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:04:21,917 INFO [INIT] Marking all workers as offline 2019-11-04 12:04:21,944 INFO [INIT] Starting encryption 2019-11-04 12:04:21,946 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:04:21,946 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:04:21,946 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:04:21,961 INFO [INIT] LAVA master has started. 2019-11-04 12:04:21,961 INFO [INIT] Using protocol version 3 2019-11-04 12:04:23,044 DEBUG lava-logs => PING(20) 2019-11-04 12:04:34,734 DEBUG Dhanu => PING(20) 2019-11-04 12:04:34,734 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:04:41,968 INFO scheduling health checks: 2019-11-04 12:04:42,023 INFO scheduling jobs: 2019-11-04 12:04:42,025 DEBUG - kvm 2019-11-04 12:04:42,097 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:04:42,097 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:04:42,105 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:04:43,419 INFO [INIT] Dropping privileges 2019-11-04 12:04:43,419 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:04:43,419 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:04:43,421 INFO [INIT] Marking all workers as offline 2019-11-04 12:04:43,446 INFO [INIT] Starting encryption 2019-11-04 12:04:43,448 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:04:43,448 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:04:43,449 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:04:43,464 INFO [INIT] LAVA master has started. 2019-11-04 12:04:43,464 INFO [INIT] Using protocol version 3 2019-11-04 12:04:43,504 DEBUG lava-logs => PING(20) 2019-11-04 12:04:54,755 DEBUG Dhanu => PING(20) 2019-11-04 12:04:54,755 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:05:03,473 INFO scheduling health checks: 2019-11-04 12:05:03,525 INFO scheduling jobs: 2019-11-04 12:05:03,527 DEBUG - kvm 2019-11-04 12:05:03,604 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:05:03,604 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:05:03,609 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:05:04,904 INFO [INIT] Dropping privileges 2019-11-04 12:05:04,904 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:05:04,904 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:05:04,906 INFO [INIT] Marking all workers as offline 2019-11-04 12:05:04,937 INFO [INIT] Starting encryption 2019-11-04 12:05:04,939 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:05:04,940 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:05:04,940 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:05:04,954 INFO [INIT] LAVA master has started. 2019-11-04 12:05:04,954 INFO [INIT] Using protocol version 3 2019-11-04 12:05:14,777 DEBUG Dhanu => PING(20) 2019-11-04 12:05:14,777 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:05:23,544 DEBUG lava-logs => PING(20) 2019-11-04 12:05:24,956 INFO scheduling health checks: 2019-11-04 12:05:25,005 INFO scheduling jobs: 2019-11-04 12:05:25,007 DEBUG - kvm 2019-11-04 12:05:25,078 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:05:25,079 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:05:25,083 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:05:26,398 INFO [INIT] Dropping privileges 2019-11-04 12:05:26,399 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:05:26,399 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:05:26,401 INFO [INIT] Marking all workers as offline 2019-11-04 12:05:26,428 INFO [INIT] Starting encryption 2019-11-04 12:05:26,430 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:05:26,431 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:05:26,431 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:05:26,445 INFO [INIT] LAVA master has started. 2019-11-04 12:05:26,446 INFO [INIT] Using protocol version 3 2019-11-04 12:05:34,795 DEBUG Dhanu => PING(20) 2019-11-04 12:05:34,796 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:05:43,565 DEBUG lava-logs => PING(20) 2019-11-04 12:05:46,448 INFO scheduling health checks: 2019-11-04 12:05:46,529 INFO scheduling jobs: 2019-11-04 12:05:46,531 DEBUG - kvm 2019-11-04 12:05:46,597 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:05:46,597 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:05:46,602 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:05:47,937 INFO [INIT] Dropping privileges 2019-11-04 12:05:47,937 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:05:47,938 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:05:47,939 INFO [INIT] Marking all workers as offline 2019-11-04 12:05:47,997 INFO [INIT] Starting encryption 2019-11-04 12:05:47,999 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:05:48,000 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:05:48,000 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:05:48,015 INFO [INIT] LAVA master has started. 2019-11-04 12:05:48,015 INFO [INIT] Using protocol version 3 2019-11-04 12:05:54,814 DEBUG Dhanu => PING(20) 2019-11-04 12:05:54,814 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:06:03,583 DEBUG lava-logs => PING(20) 2019-11-04 12:06:08,020 INFO scheduling health checks: 2019-11-04 12:06:08,076 INFO scheduling jobs: 2019-11-04 12:06:08,077 DEBUG - kvm 2019-11-04 12:06:08,142 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:06:08,143 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:06:08,157 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:06:09,415 INFO [INIT] Dropping privileges 2019-11-04 12:06:09,415 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:06:09,415 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:06:09,417 INFO [INIT] Marking all workers as offline 2019-11-04 12:06:09,444 INFO [INIT] Starting encryption 2019-11-04 12:06:09,446 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:06:09,446 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:06:09,447 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:06:09,461 INFO [INIT] LAVA master has started. 2019-11-04 12:06:09,461 INFO [INIT] Using protocol version 3 2019-11-04 12:06:14,833 DEBUG Dhanu => PING(20) 2019-11-04 12:06:14,834 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:06:23,600 DEBUG lava-logs => PING(20) 2019-11-04 12:06:29,468 INFO scheduling health checks: 2019-11-04 12:06:29,523 INFO scheduling jobs: 2019-11-04 12:06:29,524 DEBUG - kvm 2019-11-04 12:06:29,596 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:06:29,596 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:06:29,600 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:06:30,905 INFO [INIT] Dropping privileges 2019-11-04 12:06:30,905 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:06:30,905 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:06:30,907 INFO [INIT] Marking all workers as offline 2019-11-04 12:06:30,935 INFO [INIT] Starting encryption 2019-11-04 12:06:30,936 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:06:30,937 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:06:30,937 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:06:30,953 INFO [INIT] LAVA master has started. 2019-11-04 12:06:30,954 INFO [INIT] Using protocol version 3 2019-11-04 12:06:34,853 DEBUG Dhanu => PING(20) 2019-11-04 12:06:34,853 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:06:43,614 DEBUG lava-logs => PING(20) 2019-11-04 12:06:50,961 INFO scheduling health checks: 2019-11-04 12:06:51,002 INFO scheduling jobs: 2019-11-04 12:06:51,005 DEBUG - kvm 2019-11-04 12:06:51,078 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:06:51,078 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:06:51,084 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:06:52,506 INFO [INIT] Dropping privileges 2019-11-04 12:06:52,506 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:06:52,506 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:06:52,508 INFO [INIT] Marking all workers as offline 2019-11-04 12:06:52,548 INFO [INIT] Starting encryption 2019-11-04 12:06:52,550 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:06:52,550 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:06:52,551 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:06:52,565 INFO [INIT] LAVA master has started. 2019-11-04 12:06:52,566 INFO [INIT] Using protocol version 3 2019-11-04 12:06:54,870 DEBUG Dhanu => PING(20) 2019-11-04 12:06:54,871 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:07:03,626 DEBUG lava-logs => PING(20) 2019-11-04 12:07:12,574 INFO scheduling health checks: 2019-11-04 12:07:12,649 INFO scheduling jobs: 2019-11-04 12:07:12,651 DEBUG - kvm 2019-11-04 12:07:12,723 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:07:12,723 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:07:12,727 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:07:13,911 INFO [INIT] Dropping privileges 2019-11-04 12:07:13,911 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:07:13,948 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:07:13,954 INFO [INIT] Marking all workers as offline 2019-11-04 12:07:14,012 INFO [INIT] Starting encryption 2019-11-04 12:07:14,015 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:07:14,015 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:07:14,016 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:07:14,031 INFO [INIT] LAVA master has started. 2019-11-04 12:07:14,031 INFO [INIT] Using protocol version 3 2019-11-04 12:07:14,891 DEBUG Dhanu => PING(20) 2019-11-04 12:07:14,891 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:07:23,647 DEBUG lava-logs => PING(20) 2019-11-04 12:07:34,040 INFO scheduling health checks: 2019-11-04 12:07:34,107 INFO scheduling jobs: 2019-11-04 12:07:34,109 DEBUG - kvm 2019-11-04 12:07:34,180 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:07:34,180 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:07:34,188 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:07:35,440 INFO [INIT] Dropping privileges 2019-11-04 12:07:35,440 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:07:35,441 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:07:35,442 INFO [INIT] Marking all workers as offline 2019-11-04 12:07:35,463 INFO [INIT] Starting encryption 2019-11-04 12:07:35,464 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:07:35,464 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:07:35,465 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:07:35,477 INFO [INIT] LAVA master has started. 2019-11-04 12:07:35,478 INFO [INIT] Using protocol version 3 2019-11-04 12:07:35,503 DEBUG Dhanu => PING(20) 2019-11-04 12:07:35,504 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:07:43,656 DEBUG lava-logs => PING(20) 2019-11-04 12:07:55,489 INFO scheduling health checks: 2019-11-04 12:07:55,553 INFO scheduling jobs: 2019-11-04 12:07:55,555 DEBUG - kvm 2019-11-04 12:07:55,628 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:07:55,628 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:07:55,638 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:07:56,916 INFO [INIT] Dropping privileges 2019-11-04 12:07:56,916 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:07:56,916 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:07:56,918 INFO [INIT] Marking all workers as offline 2019-11-04 12:07:56,954 INFO [INIT] Starting encryption 2019-11-04 12:07:56,956 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:07:56,956 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:07:56,957 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:07:56,969 INFO [INIT] LAVA master has started. 2019-11-04 12:07:56,970 INFO [INIT] Using protocol version 3 2019-11-04 12:08:03,673 DEBUG lava-logs => PING(20) 2019-11-04 12:08:15,545 DEBUG Dhanu => PING(20) 2019-11-04 12:08:15,546 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:08:16,972 INFO scheduling health checks: 2019-11-04 12:08:17,022 INFO scheduling jobs: 2019-11-04 12:08:17,024 DEBUG - kvm 2019-11-04 12:08:17,089 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:08:17,089 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:08:17,094 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:08:18,411 INFO [INIT] Dropping privileges 2019-11-04 12:08:18,411 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:08:18,411 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:08:18,413 INFO [INIT] Marking all workers as offline 2019-11-04 12:08:18,445 INFO [INIT] Starting encryption 2019-11-04 12:08:18,446 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:08:18,446 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:08:18,447 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:08:18,459 INFO [INIT] LAVA master has started. 2019-11-04 12:08:18,460 INFO [INIT] Using protocol version 3 2019-11-04 12:08:23,691 DEBUG lava-logs => PING(20) 2019-11-04 12:08:35,566 DEBUG Dhanu => PING(20) 2019-11-04 12:08:35,567 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:08:38,463 INFO scheduling health checks: 2019-11-04 12:08:38,513 INFO scheduling jobs: 2019-11-04 12:08:38,515 DEBUG - kvm 2019-11-04 12:08:38,579 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:08:38,579 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:08:38,593 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:08:39,905 INFO [INIT] Dropping privileges 2019-11-04 12:08:39,906 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:08:39,906 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:08:39,908 INFO [INIT] Marking all workers as offline 2019-11-04 12:08:39,936 INFO [INIT] Starting encryption 2019-11-04 12:08:39,938 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:08:39,938 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:08:39,939 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:08:39,953 INFO [INIT] LAVA master has started. 2019-11-04 12:08:39,954 INFO [INIT] Using protocol version 3 2019-11-04 12:08:43,708 DEBUG lava-logs => PING(20) 2019-11-04 12:08:55,586 DEBUG Dhanu => PING(20) 2019-11-04 12:08:55,587 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:08:59,955 INFO scheduling health checks: 2019-11-04 12:09:00,004 INFO scheduling jobs: 2019-11-04 12:09:00,007 DEBUG - kvm 2019-11-04 12:09:00,077 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:09:00,078 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:09:00,082 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:09:01,585 INFO [INIT] Dropping privileges 2019-11-04 12:09:01,585 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:09:01,585 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:09:01,587 INFO [INIT] Marking all workers as offline 2019-11-04 12:09:01,638 INFO [INIT] Starting encryption 2019-11-04 12:09:01,638 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:09:01,639 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:09:01,639 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:09:01,647 INFO [INIT] LAVA master has started. 2019-11-04 12:09:01,648 INFO [INIT] Using protocol version 3 2019-11-04 12:09:03,722 DEBUG lava-logs => PING(20) 2019-11-04 12:09:15,609 DEBUG Dhanu => PING(20) 2019-11-04 12:09:15,610 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:09:21,654 INFO scheduling health checks: 2019-11-04 12:09:21,706 INFO scheduling jobs: 2019-11-04 12:09:21,708 DEBUG - kvm 2019-11-04 12:09:21,781 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:09:21,781 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:09:21,786 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:09:22,913 INFO [INIT] Dropping privileges 2019-11-04 12:09:22,913 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:09:22,913 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:09:22,915 INFO [INIT] Marking all workers as offline 2019-11-04 12:09:22,941 INFO [INIT] Starting encryption 2019-11-04 12:09:22,942 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:09:22,943 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:09:22,943 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:09:22,958 INFO [INIT] LAVA master has started. 2019-11-04 12:09:22,959 INFO [INIT] Using protocol version 3 2019-11-04 12:09:23,737 DEBUG lava-logs => PING(20) 2019-11-04 12:09:35,631 DEBUG Dhanu => PING(20) 2019-11-04 12:09:35,631 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:09:42,962 INFO scheduling health checks: 2019-11-04 12:09:43,019 INFO scheduling jobs: 2019-11-04 12:09:43,021 DEBUG - kvm 2019-11-04 12:09:43,092 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:09:43,092 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:09:43,103 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:09:44,410 INFO [INIT] Dropping privileges 2019-11-04 12:09:44,411 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:09:44,411 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:09:44,413 INFO [INIT] Marking all workers as offline 2019-11-04 12:09:44,443 INFO [INIT] Starting encryption 2019-11-04 12:09:44,445 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:09:44,446 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:09:44,446 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:09:44,461 INFO [INIT] LAVA master has started. 2019-11-04 12:09:44,462 INFO [INIT] Using protocol version 3 2019-11-04 12:09:44,554 DEBUG lava-logs => PING(20) 2019-11-04 12:09:55,652 DEBUG Dhanu => PING(20) 2019-11-04 12:09:55,652 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:10:04,470 INFO scheduling health checks: 2019-11-04 12:10:04,521 INFO scheduling jobs: 2019-11-04 12:10:04,523 DEBUG - kvm 2019-11-04 12:10:04,595 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:10:04,595 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:10:04,599 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:10:05,901 INFO [INIT] Dropping privileges 2019-11-04 12:10:05,901 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:10:05,901 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:10:05,903 INFO [INIT] Marking all workers as offline 2019-11-04 12:10:05,934 INFO [INIT] Starting encryption 2019-11-04 12:10:05,935 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:10:05,936 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:10:05,936 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:10:05,951 INFO [INIT] LAVA master has started. 2019-11-04 12:10:05,951 INFO [INIT] Using protocol version 3 2019-11-04 12:10:15,675 DEBUG Dhanu => PING(20) 2019-11-04 12:10:15,676 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:10:24,594 DEBUG lava-logs => PING(20) 2019-11-04 12:10:25,953 INFO scheduling health checks: 2019-11-04 12:10:26,001 INFO scheduling jobs: 2019-11-04 12:10:26,003 DEBUG - kvm 2019-11-04 12:10:26,076 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:10:26,076 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:10:26,085 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:10:27,405 INFO [INIT] Dropping privileges 2019-11-04 12:10:27,405 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:10:27,405 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:10:27,407 INFO [INIT] Marking all workers as offline 2019-11-04 12:10:27,435 INFO [INIT] Starting encryption 2019-11-04 12:10:27,437 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:10:27,437 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:10:27,438 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:10:27,455 INFO [INIT] LAVA master has started. 2019-11-04 12:10:27,456 INFO [INIT] Using protocol version 3 2019-11-04 12:10:35,694 DEBUG Dhanu => PING(20) 2019-11-04 12:10:35,694 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:10:44,611 DEBUG lava-logs => PING(20) 2019-11-04 12:10:47,459 INFO scheduling health checks: 2019-11-04 12:10:47,515 INFO scheduling jobs: 2019-11-04 12:10:47,516 DEBUG - kvm 2019-11-04 12:10:47,612 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:10:47,612 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/device-types/kvm01.jinja2", line 1, in top-level template code {% extends 'kvm01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:10:47,622 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:10:49,147 INFO [INIT] Dropping privileges 2019-11-04 12:10:49,147 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:10:49,147 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:10:49,149 INFO [INIT] Marking all workers as offline 2019-11-04 12:10:49,205 INFO [INIT] Starting encryption 2019-11-04 12:10:49,207 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:10:49,208 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:10:49,208 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:10:49,224 INFO [INIT] LAVA master has started. 2019-11-04 12:10:49,224 INFO [INIT] Using protocol version 3 2019-11-04 12:10:55,706 DEBUG Dhanu => PING(20) 2019-11-04 12:10:55,706 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:11:04,628 DEBUG lava-logs => PING(20) 2019-11-04 12:11:09,229 INFO scheduling health checks: 2019-11-04 12:11:09,261 INFO scheduling jobs: 2019-11-04 12:11:09,264 DEBUG - qemu 2019-11-04 12:11:09,343 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:11:09,344 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:11:09,353 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:11:10,679 INFO [INIT] Dropping privileges 2019-11-04 12:11:10,680 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:11:10,680 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:11:10,682 INFO [INIT] Marking all workers as offline 2019-11-04 12:11:10,707 INFO [INIT] Starting encryption 2019-11-04 12:11:10,709 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:11:10,709 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:11:10,710 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:11:10,723 INFO [INIT] LAVA master has started. 2019-11-04 12:11:10,724 INFO [INIT] Using protocol version 3 2019-11-04 12:11:15,724 DEBUG Dhanu => PING(20) 2019-11-04 12:11:15,724 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:11:24,638 DEBUG lava-logs => PING(20) 2019-11-04 12:11:30,730 INFO scheduling health checks: 2019-11-04 12:11:30,764 INFO scheduling jobs: 2019-11-04 12:11:30,766 DEBUG - qemu 2019-11-04 12:11:30,840 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:11:30,840 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:11:30,845 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:11:32,155 INFO [INIT] Dropping privileges 2019-11-04 12:11:32,155 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:11:32,156 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:11:32,157 INFO [INIT] Marking all workers as offline 2019-11-04 12:11:32,186 INFO [INIT] Starting encryption 2019-11-04 12:11:32,187 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:11:32,187 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:11:32,188 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:11:32,202 INFO [INIT] LAVA master has started. 2019-11-04 12:11:32,202 INFO [INIT] Using protocol version 3 2019-11-04 12:11:35,744 DEBUG Dhanu => PING(20) 2019-11-04 12:11:35,745 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:11:44,654 DEBUG lava-logs => PING(20) 2019-11-04 12:11:52,209 INFO scheduling health checks: 2019-11-04 12:11:52,255 INFO scheduling jobs: 2019-11-04 12:11:52,258 DEBUG - qemu 2019-11-04 12:11:52,330 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:11:52,330 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:11:52,335 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:11:53,663 INFO [INIT] Dropping privileges 2019-11-04 12:11:53,663 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:11:53,664 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:11:53,665 INFO [INIT] Marking all workers as offline 2019-11-04 12:11:53,689 INFO [INIT] Starting encryption 2019-11-04 12:11:53,691 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:11:53,691 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:11:53,691 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:11:53,706 INFO [INIT] LAVA master has started. 2019-11-04 12:11:53,706 INFO [INIT] Using protocol version 3 2019-11-04 12:11:55,761 DEBUG Dhanu => PING(20) 2019-11-04 12:11:55,762 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:12:04,667 DEBUG lava-logs => PING(20) 2019-11-04 12:12:13,715 INFO scheduling health checks: 2019-11-04 12:12:13,757 INFO scheduling jobs: 2019-11-04 12:12:13,759 DEBUG - qemu 2019-11-04 12:12:13,839 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:12:13,839 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:12:13,849 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:12:15,173 INFO [INIT] Dropping privileges 2019-11-04 12:12:15,173 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:12:15,174 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:12:15,175 INFO [INIT] Marking all workers as offline 2019-11-04 12:12:15,203 INFO [INIT] Starting encryption 2019-11-04 12:12:15,204 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:12:15,205 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:12:15,205 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:12:15,219 INFO [INIT] LAVA master has started. 2019-11-04 12:12:15,219 INFO [INIT] Using protocol version 3 2019-11-04 12:12:15,779 DEBUG Dhanu => PING(20) 2019-11-04 12:12:15,779 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:12:24,688 DEBUG lava-logs => PING(20) 2019-11-04 12:12:35,230 INFO scheduling health checks: 2019-11-04 12:12:35,270 INFO scheduling jobs: 2019-11-04 12:12:35,274 DEBUG - qemu 2019-11-04 12:12:35,350 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:12:35,351 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:12:35,365 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:12:36,735 INFO [INIT] Dropping privileges 2019-11-04 12:12:36,735 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:12:36,735 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:12:36,737 INFO [INIT] Marking all workers as offline 2019-11-04 12:12:36,770 INFO [INIT] Starting encryption 2019-11-04 12:12:36,771 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:12:36,771 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:12:36,772 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:12:36,783 INFO [INIT] LAVA master has started. 2019-11-04 12:12:36,783 INFO [INIT] Using protocol version 3 2019-11-04 12:12:36,883 DEBUG lava-logs => PING(20) 2019-11-04 12:12:36,903 DEBUG lava-logs => PING(20) 2019-11-04 12:12:36,936 DEBUG Dhanu => PING(20) 2019-11-04 12:12:36,936 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:12:56,801 INFO scheduling health checks: 2019-11-04 12:12:56,827 INFO scheduling jobs: 2019-11-04 12:12:56,829 DEBUG - qemu 2019-11-04 12:12:56,903 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:12:56,903 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:12:56,908 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:12:58,179 INFO [INIT] Dropping privileges 2019-11-04 12:12:58,179 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:12:58,179 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:12:58,181 INFO [INIT] Marking all workers as offline 2019-11-04 12:12:58,217 INFO [INIT] Starting encryption 2019-11-04 12:12:58,219 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:12:58,220 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:12:58,220 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:12:58,235 INFO [INIT] LAVA master has started. 2019-11-04 12:12:58,235 INFO [INIT] Using protocol version 3 2019-11-04 12:12:58,277 DEBUG lava-logs => PING(20) 2019-11-04 12:12:58,396 DEBUG Dhanu => PING(20) 2019-11-04 12:12:58,397 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:13:04,538 INFO [POLL] Received a signal, leaving 2019-11-04 12:13:04,538 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:13:05,522 INFO [INIT] Dropping privileges 2019-11-04 12:13:05,522 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:13:05,522 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:13:05,524 INFO [INIT] Marking all workers as offline 2019-11-04 12:13:05,563 INFO [INIT] Starting encryption 2019-11-04 12:13:05,564 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:13:05,565 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:13:05,566 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:13:05,583 INFO [INIT] LAVA master has started. 2019-11-04 12:13:05,584 INFO [INIT] Using protocol version 3 2019-11-04 12:13:18,291 DEBUG lava-logs => PING(20) 2019-11-04 12:13:18,415 DEBUG Dhanu => PING(20) 2019-11-04 12:13:18,416 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:13:23,503 INFO Dhanu => HELLO 2019-11-04 12:13:23,504 WARNING Dispatcher has RESTARTED 2019-11-04 12:13:25,585 INFO scheduling health checks: 2019-11-04 12:13:25,619 INFO scheduling jobs: 2019-11-04 12:13:25,622 DEBUG - qemu 2019-11-04 12:13:25,693 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:13:25,693 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:13:25,698 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:13:26,918 INFO [INIT] Dropping privileges 2019-11-04 12:13:26,919 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:13:26,919 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:13:26,921 INFO [INIT] Marking all workers as offline 2019-11-04 12:13:26,953 INFO [INIT] Starting encryption 2019-11-04 12:13:26,955 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:13:26,955 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:13:26,956 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:13:26,970 INFO [INIT] LAVA master has started. 2019-11-04 12:13:26,970 INFO [INIT] Using protocol version 3 2019-11-04 12:13:28,508 DEBUG Dhanu => PING(20) 2019-11-04 12:13:28,509 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:13:38,305 DEBUG lava-logs => PING(20) 2019-11-04 12:13:46,979 INFO scheduling health checks: 2019-11-04 12:13:47,010 INFO scheduling jobs: 2019-11-04 12:13:47,012 DEBUG - qemu 2019-11-04 12:13:47,087 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:13:47,087 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:13:47,096 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:13:48,524 INFO [INIT] Dropping privileges 2019-11-04 12:13:48,524 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:13:48,525 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:13:48,526 INFO [INIT] Marking all workers as offline 2019-11-04 12:13:48,567 INFO [INIT] Starting encryption 2019-11-04 12:13:48,570 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:13:48,571 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:13:48,571 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:13:48,585 INFO [INIT] LAVA master has started. 2019-11-04 12:13:48,586 INFO [INIT] Using protocol version 3 2019-11-04 12:13:48,588 DEBUG Dhanu => PING(20) 2019-11-04 12:13:48,588 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:13:58,322 DEBUG lava-logs => PING(20) 2019-11-04 12:14:08,596 INFO scheduling health checks: 2019-11-04 12:14:08,635 INFO scheduling jobs: 2019-11-04 12:14:08,637 DEBUG - qemu 2019-11-04 12:14:08,727 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:14:08,727 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:14:08,737 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:14:09,939 INFO [INIT] Dropping privileges 2019-11-04 12:14:09,939 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:14:09,939 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:14:09,941 INFO [INIT] Marking all workers as offline 2019-11-04 12:14:09,979 INFO [INIT] Starting encryption 2019-11-04 12:14:09,980 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:14:09,980 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:14:09,981 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:14:09,991 INFO [INIT] LAVA master has started. 2019-11-04 12:14:09,991 INFO [INIT] Using protocol version 3 2019-11-04 12:14:18,335 DEBUG lava-logs => PING(20) 2019-11-04 12:14:28,625 DEBUG Dhanu => PING(20) 2019-11-04 12:14:28,625 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:14:29,993 INFO scheduling health checks: 2019-11-04 12:14:30,026 INFO scheduling jobs: 2019-11-04 12:14:30,028 DEBUG - qemu 2019-11-04 12:14:30,104 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:14:30,105 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:14:30,113 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:14:31,395 INFO [INIT] Dropping privileges 2019-11-04 12:14:31,395 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:14:31,396 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:14:31,397 INFO [INIT] Marking all workers as offline 2019-11-04 12:14:31,427 INFO [INIT] Starting encryption 2019-11-04 12:14:31,428 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:14:31,429 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:14:31,429 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:14:31,443 INFO [INIT] LAVA master has started. 2019-11-04 12:14:31,443 INFO [INIT] Using protocol version 3 2019-11-04 12:14:38,354 DEBUG lava-logs => PING(20) 2019-11-04 12:14:48,643 DEBUG Dhanu => PING(20) 2019-11-04 12:14:48,644 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:14:51,446 INFO scheduling health checks: 2019-11-04 12:14:51,495 INFO scheduling jobs: 2019-11-04 12:14:51,498 DEBUG - qemu 2019-11-04 12:14:51,585 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:14:51,585 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:14:51,593 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:14:52,932 INFO [INIT] Dropping privileges 2019-11-04 12:14:52,932 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:14:52,932 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:14:52,934 INFO [INIT] Marking all workers as offline 2019-11-04 12:14:52,962 INFO [INIT] Starting encryption 2019-11-04 12:14:52,963 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:14:52,963 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:14:52,963 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:14:52,978 INFO [INIT] LAVA master has started. 2019-11-04 12:14:52,978 INFO [INIT] Using protocol version 3 2019-11-04 12:14:58,371 DEBUG lava-logs => PING(20) 2019-11-04 12:15:08,663 DEBUG Dhanu => PING(20) 2019-11-04 12:15:08,664 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:15:12,979 INFO scheduling health checks: 2019-11-04 12:15:13,008 INFO scheduling jobs: 2019-11-04 12:15:13,010 DEBUG - qemu 2019-11-04 12:15:13,087 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:15:13,087 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:15:13,092 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:15:14,438 INFO [INIT] Dropping privileges 2019-11-04 12:15:14,438 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:15:14,438 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:15:14,440 INFO [INIT] Marking all workers as offline 2019-11-04 12:15:14,475 INFO [INIT] Starting encryption 2019-11-04 12:15:14,478 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:15:14,479 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:15:14,479 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:15:14,494 INFO [INIT] LAVA master has started. 2019-11-04 12:15:14,495 INFO [INIT] Using protocol version 3 2019-11-04 12:15:22,235 DEBUG lava-logs => PING(20) 2019-11-04 12:15:22,264 DEBUG lava-logs => PING(20) 2019-11-04 12:15:28,684 DEBUG Dhanu => PING(20) 2019-11-04 12:15:28,685 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:15:34,500 INFO scheduling health checks: 2019-11-04 12:15:34,532 INFO scheduling jobs: 2019-11-04 12:15:34,534 DEBUG - qemu 2019-11-04 12:15:34,610 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:15:34,610 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:15:34,615 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:15:35,926 INFO [INIT] Dropping privileges 2019-11-04 12:15:35,926 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:15:35,926 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:15:35,928 INFO [INIT] Marking all workers as offline 2019-11-04 12:15:35,966 INFO [INIT] Starting encryption 2019-11-04 12:15:35,968 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:15:35,969 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:15:35,969 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:15:35,983 INFO [INIT] LAVA master has started. 2019-11-04 12:15:35,984 INFO [INIT] Using protocol version 3 2019-11-04 12:15:42,283 DEBUG lava-logs => PING(20) 2019-11-04 12:15:48,702 DEBUG Dhanu => PING(20) 2019-11-04 12:15:48,702 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:15:48,938 INFO [POLL] Received a signal, leaving 2019-11-04 12:15:48,938 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:15:49,904 INFO [INIT] Dropping privileges 2019-11-04 12:15:49,904 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:15:49,905 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:15:49,906 INFO [INIT] Marking all workers as offline 2019-11-04 12:15:49,935 INFO [INIT] Starting encryption 2019-11-04 12:15:49,936 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:15:49,937 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:15:49,937 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:15:49,951 INFO [INIT] LAVA master has started. 2019-11-04 12:15:49,951 INFO [INIT] Using protocol version 3 2019-11-04 12:15:51,640 INFO Dhanu => HELLO 2019-11-04 12:15:51,640 WARNING New dispatcher 2019-11-04 12:15:56,646 DEBUG Dhanu => PING(20) 2019-11-04 12:16:02,297 DEBUG lava-logs => PING(20) 2019-11-04 12:16:09,958 INFO scheduling health checks: 2019-11-04 12:16:10,002 INFO scheduling jobs: 2019-11-04 12:16:10,004 DEBUG - qemu 2019-11-04 12:16:10,080 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:16:10,080 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:16:10,089 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:16:11,441 INFO [INIT] Dropping privileges 2019-11-04 12:16:11,441 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:16:11,441 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:16:11,443 INFO [INIT] Marking all workers as offline 2019-11-04 12:16:11,481 INFO [INIT] Starting encryption 2019-11-04 12:16:11,482 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:16:11,482 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:16:11,483 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:16:11,497 INFO [INIT] LAVA master has started. 2019-11-04 12:16:11,497 INFO [INIT] Using protocol version 3 2019-11-04 12:16:16,659 DEBUG Dhanu => PING(20) 2019-11-04 12:16:16,660 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:16:22,311 DEBUG lava-logs => PING(20) 2019-11-04 12:16:31,507 INFO scheduling health checks: 2019-11-04 12:16:31,538 INFO scheduling jobs: 2019-11-04 12:16:31,540 DEBUG - qemu 2019-11-04 12:16:31,616 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:16:31,617 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:16:31,626 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:16:32,920 INFO [INIT] Dropping privileges 2019-11-04 12:16:32,921 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:16:32,921 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:16:32,922 INFO [INIT] Marking all workers as offline 2019-11-04 12:16:32,950 INFO [INIT] Starting encryption 2019-11-04 12:16:32,952 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:16:32,953 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:16:32,953 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:16:32,967 INFO [INIT] LAVA master has started. 2019-11-04 12:16:32,968 INFO [INIT] Using protocol version 3 2019-11-04 12:16:36,681 DEBUG Dhanu => PING(20) 2019-11-04 12:16:36,682 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:16:42,330 DEBUG lava-logs => PING(20) 2019-11-04 12:16:52,978 INFO scheduling health checks: 2019-11-04 12:16:53,029 INFO scheduling jobs: 2019-11-04 12:16:53,032 DEBUG - qemu 2019-11-04 12:16:53,110 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:16:53,110 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:16:53,114 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:16:54,446 INFO [INIT] Dropping privileges 2019-11-04 12:16:54,446 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:16:54,447 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:16:54,449 INFO [INIT] Marking all workers as offline 2019-11-04 12:16:54,508 INFO [INIT] Starting encryption 2019-11-04 12:16:54,509 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:16:54,510 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:16:54,511 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:16:54,525 INFO [INIT] LAVA master has started. 2019-11-04 12:16:54,525 INFO [INIT] Using protocol version 3 2019-11-04 12:16:56,701 DEBUG Dhanu => PING(20) 2019-11-04 12:16:56,701 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:17:02,342 DEBUG lava-logs => PING(20) 2019-11-04 12:17:14,537 INFO scheduling health checks: 2019-11-04 12:17:14,575 INFO scheduling jobs: 2019-11-04 12:17:14,579 DEBUG - qemu 2019-11-04 12:17:14,659 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:17:14,659 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:17:14,664 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:17:16,085 INFO [INIT] Dropping privileges 2019-11-04 12:17:16,085 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:17:16,086 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:17:16,087 INFO [INIT] Marking all workers as offline 2019-11-04 12:17:16,153 INFO [INIT] Starting encryption 2019-11-04 12:17:16,154 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:17:16,155 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:17:16,155 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:17:16,164 INFO [INIT] LAVA master has started. 2019-11-04 12:17:16,164 INFO [INIT] Using protocol version 3 2019-11-04 12:17:16,716 DEBUG Dhanu => PING(20) 2019-11-04 12:17:16,716 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:17:22,360 DEBUG lava-logs => PING(20) 2019-11-04 12:17:36,176 INFO scheduling health checks: 2019-11-04 12:17:36,211 INFO scheduling jobs: 2019-11-04 12:17:36,213 DEBUG - qemu 2019-11-04 12:17:36,289 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:17:36,289 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:17:36,293 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:17:37,441 INFO [INIT] Dropping privileges 2019-11-04 12:17:37,441 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:17:37,441 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:17:37,443 INFO [INIT] Marking all workers as offline 2019-11-04 12:17:37,478 INFO [INIT] Starting encryption 2019-11-04 12:17:37,480 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:17:37,480 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:17:37,481 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:17:37,493 INFO [INIT] LAVA master has started. 2019-11-04 12:17:37,494 INFO [INIT] Using protocol version 3 2019-11-04 12:17:37,637 DEBUG Dhanu => PING(20) 2019-11-04 12:17:37,637 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:17:42,374 DEBUG lava-logs => PING(20) 2019-11-04 12:17:57,497 INFO scheduling health checks: 2019-11-04 12:17:57,535 INFO scheduling jobs: 2019-11-04 12:17:57,539 DEBUG - qemu 2019-11-04 12:17:57,616 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:17:57,616 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:17:57,623 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:17:58,927 INFO [INIT] Dropping privileges 2019-11-04 12:17:58,928 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:17:58,928 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:17:58,930 INFO [INIT] Marking all workers as offline 2019-11-04 12:17:58,958 INFO [INIT] Starting encryption 2019-11-04 12:17:58,960 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:17:58,960 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:17:58,961 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:17:58,975 INFO [INIT] LAVA master has started. 2019-11-04 12:17:58,975 INFO [INIT] Using protocol version 3 2019-11-04 12:17:59,084 DEBUG Dhanu => PING(20) 2019-11-04 12:17:59,084 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:18:02,386 DEBUG lava-logs => PING(20) 2019-11-04 12:18:18,992 INFO scheduling health checks: 2019-11-04 12:18:19,037 INFO scheduling jobs: 2019-11-04 12:18:19,040 DEBUG - qemu 2019-11-04 12:18:19,118 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:18:19,118 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:18:19,122 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:18:20,412 INFO [INIT] Dropping privileges 2019-11-04 12:18:20,413 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:18:20,413 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:18:20,415 INFO [INIT] Marking all workers as offline 2019-11-04 12:18:20,449 INFO [INIT] Starting encryption 2019-11-04 12:18:20,450 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:18:20,450 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:18:20,450 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:18:20,464 INFO [INIT] LAVA master has started. 2019-11-04 12:18:20,464 INFO [INIT] Using protocol version 3 2019-11-04 12:18:22,400 DEBUG lava-logs => PING(20) 2019-11-04 12:18:39,117 DEBUG Dhanu => PING(20) 2019-11-04 12:18:39,117 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:18:40,466 INFO scheduling health checks: 2019-11-04 12:18:40,506 INFO scheduling jobs: 2019-11-04 12:18:40,508 DEBUG - qemu 2019-11-04 12:18:40,583 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:18:40,583 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:18:40,588 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:18:42,041 INFO [INIT] Dropping privileges 2019-11-04 12:18:42,042 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:18:42,042 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:18:42,044 INFO [INIT] Marking all workers as offline 2019-11-04 12:18:42,073 INFO [INIT] Starting encryption 2019-11-04 12:18:42,074 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:18:42,075 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:18:42,075 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:18:42,090 INFO [INIT] LAVA master has started. 2019-11-04 12:18:42,090 INFO [INIT] Using protocol version 3 2019-11-04 12:18:42,409 DEBUG lava-logs => PING(20) 2019-11-04 12:18:59,135 DEBUG Dhanu => PING(20) 2019-11-04 12:18:59,135 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:19:02,093 INFO scheduling health checks: 2019-11-04 12:19:02,119 INFO scheduling jobs: 2019-11-04 12:19:02,121 DEBUG - qemu 2019-11-04 12:19:02,194 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:19:02,194 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:19:02,204 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:19:03,458 INFO [INIT] Dropping privileges 2019-11-04 12:19:03,458 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:19:03,459 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:19:03,460 INFO [INIT] Marking all workers as offline 2019-11-04 12:19:03,498 INFO [INIT] Starting encryption 2019-11-04 12:19:03,499 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:19:03,499 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:19:03,499 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:19:03,513 INFO [INIT] LAVA master has started. 2019-11-04 12:19:03,513 INFO [INIT] Using protocol version 3 2019-11-04 12:19:03,537 DEBUG lava-logs => PING(20) 2019-11-04 12:19:19,153 DEBUG Dhanu => PING(20) 2019-11-04 12:19:19,153 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:19:23,516 INFO scheduling health checks: 2019-11-04 12:19:23,543 INFO scheduling jobs: 2019-11-04 12:19:23,545 DEBUG - qemu 2019-11-04 12:19:23,619 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:19:23,619 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 4, in top-level template code {% set netdevice = 'tap' %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:19:23,624 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:19:24,912 INFO [INIT] Dropping privileges 2019-11-04 12:19:24,912 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:19:24,912 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:19:24,914 INFO [INIT] Marking all workers as offline 2019-11-04 12:19:24,945 INFO [INIT] Starting encryption 2019-11-04 12:19:24,947 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:19:24,947 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:19:24,948 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:19:24,962 INFO [INIT] LAVA master has started. 2019-11-04 12:19:24,962 INFO [INIT] Using protocol version 3 2019-11-04 12:19:39,176 DEBUG Dhanu => PING(20) 2019-11-04 12:19:39,176 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:19:43,574 DEBUG lava-logs => PING(20) 2019-11-04 12:19:44,964 INFO scheduling health checks: 2019-11-04 12:19:45,001 INFO scheduling jobs: 2019-11-04 12:19:45,004 DEBUG - qemu 2019-11-04 12:19:45,074 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:19:45,075 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:19:45,079 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:19:46,148 INFO [INIT] Dropping privileges 2019-11-04 12:19:46,149 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:19:46,149 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:19:46,151 INFO [INIT] Marking all workers as offline 2019-11-04 12:19:46,180 INFO [INIT] Starting encryption 2019-11-04 12:19:46,182 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:19:46,182 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:19:46,183 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:19:46,198 INFO [INIT] LAVA master has started. 2019-11-04 12:19:46,199 INFO [INIT] Using protocol version 3 2019-11-04 12:19:59,194 DEBUG Dhanu => PING(20) 2019-11-04 12:19:59,194 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:20:06,206 WARNING lava-logs is offline: can't schedule jobs 2019-11-04 12:20:07,776 DEBUG lava-logs => PING(20) 2019-11-04 12:20:07,804 DEBUG lava-logs => PING(20) 2019-11-04 12:20:13,637 INFO Dhanu => HELLO 2019-11-04 12:20:13,637 WARNING Dispatcher has RESTARTED 2019-11-04 12:20:16,913 INFO [POLL] Received a signal, leaving 2019-11-04 12:20:16,914 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:20:17,898 INFO [INIT] Dropping privileges 2019-11-04 12:20:17,899 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:20:17,899 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:20:17,901 INFO [INIT] Marking all workers as offline 2019-11-04 12:20:17,927 INFO [INIT] Starting encryption 2019-11-04 12:20:17,928 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:20:17,928 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:20:17,928 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:20:17,941 INFO [INIT] LAVA master has started. 2019-11-04 12:20:17,941 INFO [INIT] Using protocol version 3 2019-11-04 12:20:18,640 DEBUG Dhanu => PING(20) 2019-11-04 12:20:18,641 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:20:27,826 DEBUG lava-logs => PING(20) 2019-11-04 12:20:37,948 INFO scheduling health checks: 2019-11-04 12:20:37,984 INFO scheduling jobs: 2019-11-04 12:20:37,986 DEBUG - qemu 2019-11-04 12:20:38,059 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:20:38,059 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:20:38,070 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:20:39,169 INFO [INIT] Dropping privileges 2019-11-04 12:20:39,169 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:20:39,169 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:20:39,171 INFO [INIT] Marking all workers as offline 2019-11-04 12:20:39,207 INFO [INIT] Starting encryption 2019-11-04 12:20:39,208 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:20:39,209 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:20:39,209 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:20:39,224 INFO [INIT] LAVA master has started. 2019-11-04 12:20:39,224 INFO [INIT] Using protocol version 3 2019-11-04 12:20:39,274 DEBUG Dhanu => PING(20) 2019-11-04 12:20:39,275 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:20:47,842 DEBUG lava-logs => PING(20) 2019-11-04 12:20:59,235 INFO scheduling health checks: 2019-11-04 12:20:59,264 INFO scheduling jobs: 2019-11-04 12:20:59,266 DEBUG - qemu 2019-11-04 12:20:59,337 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:20:59,338 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:20:59,342 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:21:00,656 INFO [INIT] Dropping privileges 2019-11-04 12:21:00,657 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:21:00,657 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:21:00,658 INFO [INIT] Marking all workers as offline 2019-11-04 12:21:00,698 INFO [INIT] Starting encryption 2019-11-04 12:21:00,700 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:21:00,701 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:21:00,701 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:21:00,719 INFO [INIT] LAVA master has started. 2019-11-04 12:21:00,719 INFO [INIT] Using protocol version 3 2019-11-04 12:21:07,863 DEBUG lava-logs => PING(20) 2019-11-04 12:21:19,314 DEBUG Dhanu => PING(20) 2019-11-04 12:21:19,314 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:21:20,720 INFO scheduling health checks: 2019-11-04 12:21:20,744 INFO scheduling jobs: 2019-11-04 12:21:20,746 DEBUG - qemu 2019-11-04 12:21:20,816 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:21:20,817 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:21:20,821 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:21:21,890 INFO [INIT] Dropping privileges 2019-11-04 12:21:21,890 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:21:21,891 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:21:21,892 INFO [INIT] Marking all workers as offline 2019-11-04 12:21:21,934 INFO [INIT] Starting encryption 2019-11-04 12:21:21,936 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:21:21,936 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:21:21,937 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:21:21,950 INFO [INIT] LAVA master has started. 2019-11-04 12:21:21,950 INFO [INIT] Using protocol version 3 2019-11-04 12:21:27,878 DEBUG lava-logs => PING(20) 2019-11-04 12:21:39,333 DEBUG Dhanu => PING(20) 2019-11-04 12:21:39,333 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:21:41,953 INFO scheduling health checks: 2019-11-04 12:21:41,979 INFO scheduling jobs: 2019-11-04 12:21:41,982 DEBUG - qemu 2019-11-04 12:21:42,054 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:21:42,054 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 2, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:21:42,059 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:21:43,160 INFO [INIT] Dropping privileges 2019-11-04 12:21:43,160 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:21:43,161 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:21:43,162 INFO [INIT] Marking all workers as offline 2019-11-04 12:21:43,237 INFO [INIT] Starting encryption 2019-11-04 12:21:43,238 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:21:43,239 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:21:43,239 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:21:43,251 INFO [INIT] LAVA master has started. 2019-11-04 12:21:43,251 INFO [INIT] Using protocol version 3 2019-11-04 12:21:47,895 DEBUG lava-logs => PING(20) 2019-11-04 12:21:59,353 DEBUG Dhanu => PING(20) 2019-11-04 12:21:59,353 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:22:03,151 DEBUG lava-logs => PING(20) 2019-11-04 12:22:03,204 DEBUG lava-logs => PING(20) 2019-11-04 12:22:03,252 INFO scheduling health checks: 2019-11-04 12:22:03,292 INFO scheduling jobs: 2019-11-04 12:22:03,296 DEBUG - qemu 2019-11-04 12:22:03,362 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:22:03,362 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:22:03,366 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:22:05,422 INFO [INIT] Dropping privileges 2019-11-04 12:22:05,422 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:22:05,422 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:22:05,423 INFO [INIT] Marking all workers as offline 2019-11-04 12:22:05,494 INFO [INIT] Starting encryption 2019-11-04 12:22:05,495 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:22:05,495 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:22:05,496 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:22:05,507 INFO [INIT] LAVA master has started. 2019-11-04 12:22:05,508 INFO [INIT] Using protocol version 3 2019-11-04 12:22:09,130 INFO Dhanu => HELLO 2019-11-04 12:22:09,130 WARNING New dispatcher 2019-11-04 12:22:14,137 DEBUG Dhanu => PING(20) 2019-11-04 12:22:23,218 DEBUG lava-logs => PING(20) 2019-11-04 12:22:25,510 INFO scheduling health checks: 2019-11-04 12:22:25,550 INFO scheduling jobs: 2019-11-04 12:22:25,553 DEBUG - qemu 2019-11-04 12:22:25,631 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:22:25,631 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:22:25,636 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:22:27,070 INFO [INIT] Dropping privileges 2019-11-04 12:22:27,070 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:22:27,070 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:22:27,072 INFO [INIT] Marking all workers as offline 2019-11-04 12:22:27,129 INFO [INIT] Starting encryption 2019-11-04 12:22:27,131 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:22:27,131 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:22:27,132 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:22:27,145 INFO [INIT] LAVA master has started. 2019-11-04 12:22:27,146 INFO [INIT] Using protocol version 3 2019-11-04 12:22:34,158 DEBUG Dhanu => PING(20) 2019-11-04 12:22:34,159 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:22:43,236 DEBUG lava-logs => PING(20) 2019-11-04 12:22:47,150 INFO scheduling health checks: 2019-11-04 12:22:47,186 INFO scheduling jobs: 2019-11-04 12:22:47,191 DEBUG - qemu 2019-11-04 12:22:47,273 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:22:47,273 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:22:47,283 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:22:48,448 INFO [INIT] Dropping privileges 2019-11-04 12:22:48,448 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:22:48,449 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:22:48,450 INFO [INIT] Marking all workers as offline 2019-11-04 12:22:48,488 INFO [INIT] Starting encryption 2019-11-04 12:22:48,490 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:22:48,490 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:22:48,491 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:22:48,511 INFO [INIT] LAVA master has started. 2019-11-04 12:22:48,511 INFO [INIT] Using protocol version 3 2019-11-04 12:22:54,177 DEBUG Dhanu => PING(20) 2019-11-04 12:22:54,178 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:23:03,253 DEBUG lava-logs => PING(20) 2019-11-04 12:23:08,517 INFO scheduling health checks: 2019-11-04 12:23:08,554 INFO scheduling jobs: 2019-11-04 12:23:08,558 DEBUG - qemu 2019-11-04 12:23:08,636 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:23:08,638 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:23:08,643 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:23:09,920 INFO [INIT] Dropping privileges 2019-11-04 12:23:09,920 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:23:09,920 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:23:09,922 INFO [INIT] Marking all workers as offline 2019-11-04 12:23:09,956 INFO [INIT] Starting encryption 2019-11-04 12:23:09,958 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:23:09,958 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:23:09,958 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:23:09,972 INFO [INIT] LAVA master has started. 2019-11-04 12:23:09,972 INFO [INIT] Using protocol version 3 2019-11-04 12:23:14,201 DEBUG Dhanu => PING(20) 2019-11-04 12:23:14,202 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:23:23,268 DEBUG lava-logs => PING(20) 2019-11-04 12:23:29,975 INFO scheduling health checks: 2019-11-04 12:23:30,012 INFO scheduling jobs: 2019-11-04 12:23:30,014 DEBUG - qemu 2019-11-04 12:23:30,083 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:23:30,083 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:23:30,088 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:23:31,411 INFO [INIT] Dropping privileges 2019-11-04 12:23:31,411 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:23:31,412 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:23:31,413 INFO [INIT] Marking all workers as offline 2019-11-04 12:23:31,446 INFO [INIT] Starting encryption 2019-11-04 12:23:31,449 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:23:31,450 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:23:31,450 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:23:31,463 INFO [INIT] LAVA master has started. 2019-11-04 12:23:31,463 INFO [INIT] Using protocol version 3 2019-11-04 12:23:34,223 DEBUG Dhanu => PING(20) 2019-11-04 12:23:34,224 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:23:43,283 DEBUG lava-logs => PING(20) 2019-11-04 12:23:51,472 INFO scheduling health checks: 2019-11-04 12:23:51,503 INFO scheduling jobs: 2019-11-04 12:23:51,506 DEBUG - qemu 2019-11-04 12:23:51,574 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:23:51,574 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:23:51,579 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:23:52,638 INFO [INIT] Dropping privileges 2019-11-04 12:23:52,638 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:23:52,638 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:23:52,640 INFO [INIT] Marking all workers as offline 2019-11-04 12:23:52,671 INFO [INIT] Starting encryption 2019-11-04 12:23:52,673 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:23:52,673 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:23:52,674 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:23:52,687 INFO [INIT] LAVA master has started. 2019-11-04 12:23:52,687 INFO [INIT] Using protocol version 3 2019-11-04 12:23:54,245 DEBUG Dhanu => PING(20) 2019-11-04 12:23:54,245 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:24:03,293 DEBUG lava-logs => PING(20) 2019-11-04 12:24:12,697 INFO scheduling health checks: 2019-11-04 12:24:12,739 INFO scheduling jobs: 2019-11-04 12:24:12,741 DEBUG - qemu 2019-11-04 12:24:12,811 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:24:12,811 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:24:12,821 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:24:13,926 INFO [INIT] Dropping privileges 2019-11-04 12:24:13,926 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:24:13,927 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:24:13,928 INFO [INIT] Marking all workers as offline 2019-11-04 12:24:13,961 INFO [INIT] Starting encryption 2019-11-04 12:24:13,962 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:24:13,963 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:24:13,963 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:24:13,978 INFO [INIT] LAVA master has started. 2019-11-04 12:24:13,978 INFO [INIT] Using protocol version 3 2019-11-04 12:24:14,264 DEBUG Dhanu => PING(20) 2019-11-04 12:24:14,264 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:24:23,312 DEBUG lava-logs => PING(20) 2019-11-04 12:24:33,985 INFO scheduling health checks: 2019-11-04 12:24:34,018 INFO scheduling jobs: 2019-11-04 12:24:34,021 DEBUG - qemu 2019-11-04 12:24:34,088 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:24:34,088 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:24:34,093 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:24:35,396 INFO [INIT] Dropping privileges 2019-11-04 12:24:35,396 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:24:35,396 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:24:35,398 INFO [INIT] Marking all workers as offline 2019-11-04 12:24:35,430 INFO [INIT] Starting encryption 2019-11-04 12:24:35,431 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:24:35,432 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:24:35,432 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:24:35,447 INFO [INIT] LAVA master has started. 2019-11-04 12:24:35,447 INFO [INIT] Using protocol version 3 2019-11-04 12:24:35,593 DEBUG Dhanu => PING(20) 2019-11-04 12:24:35,594 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:24:43,332 DEBUG lava-logs => PING(20) 2019-11-04 12:24:55,460 INFO scheduling health checks: 2019-11-04 12:24:55,498 INFO scheduling jobs: 2019-11-04 12:24:55,501 DEBUG - qemu 2019-11-04 12:24:55,572 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:24:55,573 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:24:55,582 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:24:56,912 INFO [INIT] Dropping privileges 2019-11-04 12:24:56,913 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:24:56,913 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:24:56,915 INFO [INIT] Marking all workers as offline 2019-11-04 12:24:56,955 INFO [INIT] Starting encryption 2019-11-04 12:24:56,957 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:24:56,958 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:24:56,958 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:24:56,974 INFO [INIT] LAVA master has started. 2019-11-04 12:24:56,975 INFO [INIT] Using protocol version 3 2019-11-04 12:24:56,987 DEBUG Dhanu => PING(20) 2019-11-04 12:24:56,987 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:25:03,351 DEBUG lava-logs => PING(20) 2019-11-04 12:25:16,988 INFO scheduling health checks: 2019-11-04 12:25:17,023 INFO scheduling jobs: 2019-11-04 12:25:17,025 DEBUG - qemu 2019-11-04 12:25:17,092 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:25:17,092 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:25:17,106 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:25:18,390 INFO [INIT] Dropping privileges 2019-11-04 12:25:18,390 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:25:18,390 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:25:18,392 INFO [INIT] Marking all workers as offline 2019-11-04 12:25:18,424 INFO [INIT] Starting encryption 2019-11-04 12:25:18,426 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:25:18,427 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:25:18,427 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:25:18,443 INFO [INIT] LAVA master has started. 2019-11-04 12:25:18,443 INFO [INIT] Using protocol version 3 2019-11-04 12:25:23,369 DEBUG lava-logs => PING(20) 2019-11-04 12:25:37,025 DEBUG Dhanu => PING(20) 2019-11-04 12:25:37,026 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:25:38,444 INFO scheduling health checks: 2019-11-04 12:25:38,469 INFO scheduling jobs: 2019-11-04 12:25:38,470 DEBUG - qemu 2019-11-04 12:25:38,528 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:25:38,529 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:25:38,533 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:25:39,646 INFO [INIT] Dropping privileges 2019-11-04 12:25:39,646 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:25:39,647 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:25:39,648 INFO [INIT] Marking all workers as offline 2019-11-04 12:25:39,715 INFO [INIT] Starting encryption 2019-11-04 12:25:39,717 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:25:39,717 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:25:39,718 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:25:39,731 INFO [INIT] LAVA master has started. 2019-11-04 12:25:39,731 INFO [INIT] Using protocol version 3 2019-11-04 12:25:43,385 DEBUG lava-logs => PING(20) 2019-11-04 12:25:57,042 DEBUG Dhanu => PING(20) 2019-11-04 12:25:57,042 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:25:59,734 INFO scheduling health checks: 2019-11-04 12:25:59,760 INFO scheduling jobs: 2019-11-04 12:25:59,762 DEBUG - qemu 2019-11-04 12:25:59,830 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:25:59,830 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:25:59,835 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:26:00,958 INFO [INIT] Dropping privileges 2019-11-04 12:26:00,958 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:26:00,959 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:26:00,960 INFO [INIT] Marking all workers as offline 2019-11-04 12:26:00,994 INFO [INIT] Starting encryption 2019-11-04 12:26:00,998 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:26:00,998 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:26:00,998 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:26:01,015 INFO [INIT] LAVA master has started. 2019-11-04 12:26:01,015 INFO [INIT] Using protocol version 3 2019-11-04 12:26:03,400 DEBUG lava-logs => PING(20) 2019-11-04 12:26:17,063 DEBUG Dhanu => PING(20) 2019-11-04 12:26:17,064 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:26:21,016 INFO scheduling health checks: 2019-11-04 12:26:21,040 INFO scheduling jobs: 2019-11-04 12:26:21,042 DEBUG - qemu 2019-11-04 12:26:21,109 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:26:21,109 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:26:21,119 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:26:22,450 INFO [INIT] Dropping privileges 2019-11-04 12:26:22,450 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:26:22,450 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:26:22,452 INFO [INIT] Marking all workers as offline 2019-11-04 12:26:22,485 INFO [INIT] Starting encryption 2019-11-04 12:26:22,486 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:26:22,487 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:26:22,487 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:26:22,503 INFO [INIT] LAVA master has started. 2019-11-04 12:26:22,503 INFO [INIT] Using protocol version 3 2019-11-04 12:26:23,415 DEBUG lava-logs => PING(20) 2019-11-04 12:26:37,085 DEBUG Dhanu => PING(20) 2019-11-04 12:26:37,085 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:26:42,509 INFO scheduling health checks: 2019-11-04 12:26:42,542 INFO scheduling jobs: 2019-11-04 12:26:42,545 DEBUG - qemu 2019-11-04 12:26:42,615 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:26:42,615 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:26:42,629 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:26:43,925 INFO [INIT] Dropping privileges 2019-11-04 12:26:43,925 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:26:43,926 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:26:43,927 INFO [INIT] Marking all workers as offline 2019-11-04 12:26:43,999 INFO [INIT] Starting encryption 2019-11-04 12:26:44,001 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:26:44,002 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:26:44,002 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:26:44,023 INFO [INIT] LAVA master has started. 2019-11-04 12:26:44,023 INFO [INIT] Using protocol version 3 2019-11-04 12:26:44,203 DEBUG lava-logs => PING(20) 2019-11-04 12:26:57,103 DEBUG Dhanu => PING(20) 2019-11-04 12:26:57,103 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:27:04,029 INFO scheduling health checks: 2019-11-04 12:27:04,100 INFO scheduling jobs: 2019-11-04 12:27:04,103 DEBUG - qemu 2019-11-04 12:27:04,170 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:27:04,170 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:27:04,180 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:27:05,439 INFO [INIT] Dropping privileges 2019-11-04 12:27:05,439 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:27:05,439 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:27:05,441 INFO [INIT] Marking all workers as offline 2019-11-04 12:27:05,501 INFO [INIT] Starting encryption 2019-11-04 12:27:05,503 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:27:05,503 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:27:05,504 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:27:05,520 INFO [INIT] LAVA master has started. 2019-11-04 12:27:05,520 INFO [INIT] Using protocol version 3 2019-11-04 12:27:05,552 DEBUG lava-logs => PING(20) 2019-11-04 12:27:17,122 DEBUG Dhanu => PING(20) 2019-11-04 12:27:17,123 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:27:25,528 INFO scheduling health checks: 2019-11-04 12:27:25,569 INFO scheduling jobs: 2019-11-04 12:27:25,571 DEBUG - qemu 2019-11-04 12:27:25,639 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:27:25,639 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:27:25,646 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:27:26,904 INFO [INIT] Dropping privileges 2019-11-04 12:27:26,904 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:27:26,904 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:27:26,906 INFO [INIT] Marking all workers as offline 2019-11-04 12:27:26,936 INFO [INIT] Starting encryption 2019-11-04 12:27:26,937 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:27:26,938 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:27:26,938 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:27:26,950 INFO [INIT] LAVA master has started. 2019-11-04 12:27:26,950 INFO [INIT] Using protocol version 3 2019-11-04 12:27:37,142 DEBUG Dhanu => PING(20) 2019-11-04 12:27:37,142 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:27:45,587 DEBUG lava-logs => PING(20) 2019-11-04 12:27:46,952 INFO scheduling health checks: 2019-11-04 12:27:46,971 INFO scheduling jobs: 2019-11-04 12:27:46,972 DEBUG - qemu 2019-11-04 12:27:47,029 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:27:47,030 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:27:47,034 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:27:48,210 INFO [INIT] Dropping privileges 2019-11-04 12:27:48,211 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:27:48,211 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:27:48,212 INFO [INIT] Marking all workers as offline 2019-11-04 12:27:48,238 INFO [INIT] Starting encryption 2019-11-04 12:27:48,239 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:27:48,239 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:27:48,240 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:27:48,253 INFO [INIT] LAVA master has started. 2019-11-04 12:27:48,254 INFO [INIT] Using protocol version 3 2019-11-04 12:27:57,163 DEBUG Dhanu => PING(20) 2019-11-04 12:27:57,164 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:28:05,606 DEBUG lava-logs => PING(20) 2019-11-04 12:28:08,257 INFO scheduling health checks: 2019-11-04 12:28:08,295 INFO scheduling jobs: 2019-11-04 12:28:08,297 DEBUG - qemu 2019-11-04 12:28:08,363 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:28:08,364 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:28:08,368 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:28:09,655 INFO [INIT] Dropping privileges 2019-11-04 12:28:09,655 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:28:09,656 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:28:09,657 INFO [INIT] Marking all workers as offline 2019-11-04 12:28:09,697 INFO [INIT] Starting encryption 2019-11-04 12:28:09,698 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:28:09,699 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:28:09,699 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:28:09,713 INFO [INIT] LAVA master has started. 2019-11-04 12:28:09,714 INFO [INIT] Using protocol version 3 2019-11-04 12:28:17,183 DEBUG Dhanu => PING(20) 2019-11-04 12:28:17,184 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:28:25,622 DEBUG lava-logs => PING(20) 2019-11-04 12:28:29,718 INFO scheduling health checks: 2019-11-04 12:28:29,742 INFO scheduling jobs: 2019-11-04 12:28:29,743 DEBUG - qemu 2019-11-04 12:28:29,808 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:28:29,808 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:28:29,813 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:28:30,922 INFO [INIT] Dropping privileges 2019-11-04 12:28:30,922 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:28:30,923 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:28:30,924 INFO [INIT] Marking all workers as offline 2019-11-04 12:28:30,965 INFO [INIT] Starting encryption 2019-11-04 12:28:30,967 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:28:30,967 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:28:30,968 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:28:30,979 INFO [INIT] LAVA master has started. 2019-11-04 12:28:30,979 INFO [INIT] Using protocol version 3 2019-11-04 12:28:37,203 DEBUG Dhanu => PING(20) 2019-11-04 12:28:37,203 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:28:45,633 DEBUG lava-logs => PING(20) 2019-11-04 12:28:50,985 INFO scheduling health checks: 2019-11-04 12:28:51,022 INFO scheduling jobs: 2019-11-04 12:28:51,025 DEBUG - qemu 2019-11-04 12:28:51,092 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:28:51,092 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:28:51,097 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:28:52,404 INFO [INIT] Dropping privileges 2019-11-04 12:28:52,405 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:28:52,405 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:28:52,406 INFO [INIT] Marking all workers as offline 2019-11-04 12:28:52,445 INFO [INIT] Starting encryption 2019-11-04 12:28:52,447 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:28:52,447 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:28:52,448 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:28:52,461 INFO [INIT] LAVA master has started. 2019-11-04 12:28:52,462 INFO [INIT] Using protocol version 3 2019-11-04 12:28:57,227 DEBUG Dhanu => PING(20) 2019-11-04 12:28:57,227 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:29:05,648 DEBUG lava-logs => PING(20) 2019-11-04 12:29:12,469 INFO scheduling health checks: 2019-11-04 12:29:12,501 INFO scheduling jobs: 2019-11-04 12:29:12,503 DEBUG - qemu 2019-11-04 12:29:12,563 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:29:12,563 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:29:12,568 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:29:13,646 INFO [INIT] Dropping privileges 2019-11-04 12:29:13,646 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:29:13,646 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:29:13,648 INFO [INIT] Marking all workers as offline 2019-11-04 12:29:13,680 INFO [INIT] Starting encryption 2019-11-04 12:29:13,682 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:29:13,682 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:29:13,683 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:29:13,700 INFO [INIT] LAVA master has started. 2019-11-04 12:29:13,701 INFO [INIT] Using protocol version 3 2019-11-04 12:29:17,246 DEBUG Dhanu => PING(20) 2019-11-04 12:29:17,247 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:29:25,656 DEBUG lava-logs => PING(20) 2019-11-04 12:29:33,709 INFO scheduling health checks: 2019-11-04 12:29:33,748 INFO scheduling jobs: 2019-11-04 12:29:33,751 DEBUG - qemu 2019-11-04 12:29:33,819 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:29:33,819 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:29:33,824 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:29:34,913 INFO [INIT] Dropping privileges 2019-11-04 12:29:34,913 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:29:34,914 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:29:34,915 INFO [INIT] Marking all workers as offline 2019-11-04 12:29:34,971 INFO [INIT] Starting encryption 2019-11-04 12:29:34,971 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:29:34,972 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:29:34,972 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:29:34,979 INFO [INIT] LAVA master has started. 2019-11-04 12:29:34,980 INFO [INIT] Using protocol version 3 2019-11-04 12:29:37,266 DEBUG Dhanu => PING(20) 2019-11-04 12:29:37,266 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:29:45,670 DEBUG lava-logs => PING(20) 2019-11-04 12:29:54,989 INFO scheduling health checks: 2019-11-04 12:29:55,028 INFO scheduling jobs: 2019-11-04 12:29:55,031 DEBUG - qemu 2019-11-04 12:29:55,099 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:29:55,099 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:29:55,113 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:29:56,412 INFO [INIT] Dropping privileges 2019-11-04 12:29:56,412 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:29:56,413 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:29:56,414 INFO [INIT] Marking all workers as offline 2019-11-04 12:29:56,451 INFO [INIT] Starting encryption 2019-11-04 12:29:56,453 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:29:56,454 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:29:56,454 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:29:56,471 INFO [INIT] LAVA master has started. 2019-11-04 12:29:56,472 INFO [INIT] Using protocol version 3 2019-11-04 12:29:57,286 DEBUG Dhanu => PING(20) 2019-11-04 12:29:57,286 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:30:05,692 DEBUG lava-logs => PING(20) 2019-11-04 12:30:16,483 INFO scheduling health checks: 2019-11-04 12:30:16,519 INFO scheduling jobs: 2019-11-04 12:30:16,521 DEBUG - qemu 2019-11-04 12:30:16,587 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:30:16,587 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/devices/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:30:16,592 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:30:17,896 INFO [INIT] Dropping privileges 2019-11-04 12:30:17,896 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:30:17,896 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:30:17,898 INFO [INIT] Marking all workers as offline 2019-11-04 12:30:17,931 INFO [INIT] Starting encryption 2019-11-04 12:30:17,932 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:30:17,933 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:30:17,933 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:30:17,950 INFO [INIT] LAVA master has started. 2019-11-04 12:30:17,950 INFO [INIT] Using protocol version 3 2019-11-04 12:30:17,951 DEBUG Dhanu => PING(20) 2019-11-04 12:30:17,951 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:30:25,713 DEBUG lava-logs => PING(20) 2019-11-04 12:30:37,963 INFO scheduling health checks: 2019-11-04 12:30:37,988 INFO scheduling jobs: 2019-11-04 12:30:37,989 DEBUG - qemu 2019-11-04 12:30:38,064 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:30:38,064 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:30:38,069 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:30:39,179 INFO [INIT] Dropping privileges 2019-11-04 12:30:39,179 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:30:39,179 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:30:39,181 INFO [INIT] Marking all workers as offline 2019-11-04 12:30:39,211 INFO [INIT] Starting encryption 2019-11-04 12:30:39,212 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:30:39,213 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:30:39,213 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:30:39,227 INFO [INIT] LAVA master has started. 2019-11-04 12:30:39,228 INFO [INIT] Using protocol version 3 2019-11-04 12:30:54,099 DEBUG lava-logs => PING(20) 2019-11-04 12:30:54,123 DEBUG lava-logs => PING(20) 2019-11-04 12:30:57,445 INFO [POLL] Received a signal, leaving 2019-11-04 12:30:57,446 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:30:58,428 INFO [INIT] Dropping privileges 2019-11-04 12:30:58,428 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:30:58,428 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:30:58,430 INFO [INIT] Marking all workers as offline 2019-11-04 12:30:58,501 INFO [INIT] Starting encryption 2019-11-04 12:30:58,502 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:30:58,502 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:30:58,503 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:30:58,518 INFO [INIT] LAVA master has started. 2019-11-04 12:30:58,519 INFO [INIT] Using protocol version 3 2019-11-04 12:30:58,655 DEBUG Dhanu => PING(20) 2019-11-04 12:30:58,656 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:31:07,996 INFO Dhanu => HELLO 2019-11-04 12:31:07,997 WARNING Dispatcher has RESTARTED 2019-11-04 12:31:13,001 DEBUG Dhanu => PING(20) 2019-11-04 12:31:14,137 DEBUG lava-logs => PING(20) 2019-11-04 12:31:18,521 INFO scheduling health checks: 2019-11-04 12:31:18,603 INFO scheduling jobs: 2019-11-04 12:31:18,607 DEBUG - qemu 2019-11-04 12:31:18,685 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:31:18,685 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:31:18,693 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:31:20,062 INFO [INIT] Dropping privileges 2019-11-04 12:31:20,063 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:31:20,063 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:31:20,065 INFO [INIT] Marking all workers as offline 2019-11-04 12:31:20,116 INFO [INIT] Starting encryption 2019-11-04 12:31:20,117 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:31:20,117 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:31:20,118 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:31:20,132 INFO [INIT] LAVA master has started. 2019-11-04 12:31:20,133 INFO [INIT] Using protocol version 3 2019-11-04 12:31:33,017 DEBUG Dhanu => PING(20) 2019-11-04 12:31:33,018 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:31:34,150 DEBUG lava-logs => PING(20) 2019-11-04 12:31:40,137 INFO scheduling health checks: 2019-11-04 12:31:40,171 INFO scheduling jobs: 2019-11-04 12:31:40,173 DEBUG - qemu 2019-11-04 12:31:40,272 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:31:40,272 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:31:40,283 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:31:41,458 INFO [INIT] Dropping privileges 2019-11-04 12:31:41,458 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:31:41,459 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:31:41,460 INFO [INIT] Marking all workers as offline 2019-11-04 12:31:41,495 INFO [INIT] Starting encryption 2019-11-04 12:31:41,497 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:31:41,497 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:31:41,498 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:31:41,512 INFO [INIT] LAVA master has started. 2019-11-04 12:31:41,512 INFO [INIT] Using protocol version 3 2019-11-04 12:31:53,037 DEBUG Dhanu => PING(20) 2019-11-04 12:31:53,038 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:31:54,164 DEBUG lava-logs => PING(20) 2019-11-04 12:32:01,516 INFO scheduling health checks: 2019-11-04 12:32:01,551 INFO scheduling jobs: 2019-11-04 12:32:01,555 DEBUG - qemu 2019-11-04 12:32:01,632 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:32:01,632 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:32:01,637 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:32:02,964 INFO [INIT] Dropping privileges 2019-11-04 12:32:02,964 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:32:02,965 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:32:02,966 INFO [INIT] Marking all workers as offline 2019-11-04 12:32:02,997 INFO [INIT] Starting encryption 2019-11-04 12:32:02,999 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:32:02,999 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:32:03,000 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:32:03,010 INFO [INIT] LAVA master has started. 2019-11-04 12:32:03,010 INFO [INIT] Using protocol version 3 2019-11-04 12:32:13,060 DEBUG Dhanu => PING(20) 2019-11-04 12:32:13,061 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:32:14,178 DEBUG lava-logs => PING(20) 2019-11-04 12:32:23,019 INFO scheduling health checks: 2019-11-04 12:32:23,076 INFO scheduling jobs: 2019-11-04 12:32:23,079 DEBUG - qemu 2019-11-04 12:32:23,168 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:32:23,168 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:32:23,175 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:32:24,561 INFO [INIT] Dropping privileges 2019-11-04 12:32:24,561 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:32:24,561 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:32:24,563 INFO [INIT] Marking all workers as offline 2019-11-04 12:32:24,599 INFO [INIT] Starting encryption 2019-11-04 12:32:24,600 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:32:24,601 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:32:24,601 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:32:24,619 INFO [INIT] LAVA master has started. 2019-11-04 12:32:24,619 INFO [INIT] Using protocol version 3 2019-11-04 12:32:33,079 DEBUG Dhanu => PING(20) 2019-11-04 12:32:33,080 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:32:34,198 DEBUG lava-logs => PING(20) 2019-11-04 12:32:44,629 INFO scheduling health checks: 2019-11-04 12:32:44,667 INFO scheduling jobs: 2019-11-04 12:32:44,670 DEBUG - qemu 2019-11-04 12:32:44,745 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:32:44,745 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:32:44,753 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:32:45,900 INFO [INIT] Dropping privileges 2019-11-04 12:32:45,901 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:32:45,901 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:32:45,903 INFO [INIT] Marking all workers as offline 2019-11-04 12:32:45,934 INFO [INIT] Starting encryption 2019-11-04 12:32:45,936 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:32:45,937 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:32:45,937 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:32:45,952 INFO [INIT] LAVA master has started. 2019-11-04 12:32:45,952 INFO [INIT] Using protocol version 3 2019-11-04 12:32:53,101 DEBUG Dhanu => PING(20) 2019-11-04 12:32:53,102 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:32:54,218 DEBUG lava-logs => PING(20) 2019-11-04 12:33:05,964 INFO scheduling health checks: 2019-11-04 12:33:06,002 INFO scheduling jobs: 2019-11-04 12:33:06,005 DEBUG - qemu 2019-11-04 12:33:06,075 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 12:33:06,075 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 2, in top-level template code {% set mac_addr = 'DE:AD:BE:EF:28:01' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 12:33:06,080 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:33:07,404 INFO [INIT] Dropping privileges 2019-11-04 12:33:07,404 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:33:07,405 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:33:07,406 INFO [INIT] Marking all workers as offline 2019-11-04 12:33:07,437 INFO [INIT] Starting encryption 2019-11-04 12:33:07,438 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:33:07,439 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:33:07,439 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:33:07,453 INFO [INIT] LAVA master has started. 2019-11-04 12:33:07,453 INFO [INIT] Using protocol version 3 2019-11-04 12:33:13,121 DEBUG Dhanu => PING(20) 2019-11-04 12:33:13,122 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:33:14,236 DEBUG lava-logs => PING(20) 2019-11-04 12:33:27,465 INFO scheduling health checks: 2019-11-04 12:33:27,504 INFO scheduling jobs: 2019-11-04 12:33:27,506 DEBUG - qemu 2019-11-04 12:33:27,573 DEBUG Good ??? Bad (Invalid device configuration for qemu01) 2019-11-04 12:33:27,585 DEBUG Bad ??? Bad (Invalid device configuration for qemu01) 2019-11-04 12:33:33,143 DEBUG Dhanu => PING(20) 2019-11-04 12:33:34,256 DEBUG lava-logs => PING(20) 2019-11-04 12:33:47,618 INFO scheduling health checks: 2019-11-04 12:33:47,630 INFO scheduling jobs: 2019-11-04 12:33:53,165 DEBUG Dhanu => PING(20) 2019-11-04 12:34:04,530 DEBUG lava-logs => PING(20) 2019-11-04 12:34:04,552 DEBUG lava-logs => PING(20) 2019-11-04 12:34:07,639 INFO scheduling health checks: 2019-11-04 12:34:07,649 INFO scheduling jobs: 2019-11-04 12:34:09,831 INFO Dhanu => HELLO 2019-11-04 12:34:09,831 WARNING Dispatcher has RESTARTED 2019-11-04 12:34:14,838 DEBUG Dhanu => PING(20) 2019-11-04 12:34:18,770 INFO [POLL] Received a signal, leaving 2019-11-04 12:34:18,770 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:34:19,750 INFO [INIT] Dropping privileges 2019-11-04 12:34:19,750 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:34:19,751 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:34:19,752 INFO [INIT] Marking all workers as offline 2019-11-04 12:34:19,777 INFO [INIT] Starting encryption 2019-11-04 12:34:19,778 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:34:19,778 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:34:19,779 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:34:19,791 INFO [INIT] LAVA master has started. 2019-11-04 12:34:19,792 INFO [INIT] Using protocol version 3 2019-11-04 12:34:24,566 DEBUG lava-logs => PING(20) 2019-11-04 12:34:34,853 DEBUG Dhanu => PING(20) 2019-11-04 12:34:34,853 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:34:39,797 INFO scheduling health checks: 2019-11-04 12:34:39,812 INFO scheduling jobs: 2019-11-04 12:34:44,589 DEBUG lava-logs => PING(20) 2019-11-04 12:34:54,874 DEBUG Dhanu => PING(20) 2019-11-04 12:34:59,823 INFO scheduling health checks: 2019-11-04 12:34:59,833 INFO scheduling jobs: 2019-11-04 12:35:04,612 DEBUG lava-logs => PING(20) 2019-11-04 12:35:14,897 DEBUG Dhanu => PING(20) 2019-11-04 12:35:19,843 INFO scheduling health checks: 2019-11-04 12:35:19,854 INFO scheduling jobs: 2019-11-04 12:35:24,634 DEBUG lava-logs => PING(20) 2019-11-04 12:35:34,918 DEBUG Dhanu => PING(20) 2019-11-04 12:35:39,865 INFO scheduling health checks: 2019-11-04 12:35:39,875 INFO scheduling jobs: 2019-11-04 12:35:44,657 DEBUG lava-logs => PING(20) 2019-11-04 12:35:54,935 DEBUG Dhanu => PING(20) 2019-11-04 12:35:59,889 INFO scheduling health checks: 2019-11-04 12:35:59,901 INFO scheduling jobs: 2019-11-04 12:36:04,680 DEBUG lava-logs => PING(20) 2019-11-04 12:36:14,958 DEBUG Dhanu => PING(20) 2019-11-04 12:36:19,916 INFO scheduling health checks: 2019-11-04 12:36:19,927 INFO scheduling jobs: 2019-11-04 12:36:24,703 DEBUG lava-logs => PING(20) 2019-11-04 12:36:34,979 DEBUG Dhanu => PING(20) 2019-11-04 12:36:39,939 INFO scheduling health checks: 2019-11-04 12:36:39,947 INFO scheduling jobs: 2019-11-04 12:36:44,724 DEBUG lava-logs => PING(20) 2019-11-04 12:36:55,002 DEBUG Dhanu => PING(20) 2019-11-04 12:36:59,959 INFO scheduling health checks: 2019-11-04 12:36:59,970 INFO scheduling jobs: 2019-11-04 12:37:04,741 DEBUG lava-logs => PING(20) 2019-11-04 12:37:15,016 DEBUG Dhanu => PING(20) 2019-11-04 12:37:19,983 INFO scheduling health checks: 2019-11-04 12:37:19,993 INFO scheduling jobs: 2019-11-04 12:37:24,764 DEBUG lava-logs => PING(20) 2019-11-04 12:37:35,037 DEBUG Dhanu => PING(20) 2019-11-04 12:37:40,005 INFO scheduling health checks: 2019-11-04 12:37:40,015 INFO scheduling jobs: 2019-11-04 12:37:44,782 DEBUG lava-logs => PING(20) 2019-11-04 12:37:55,054 DEBUG Dhanu => PING(20) 2019-11-04 12:38:00,029 INFO scheduling health checks: 2019-11-04 12:38:00,033 INFO scheduling jobs: 2019-11-04 12:38:00,791 DEBUG lava-logs => PING(20) 2019-11-04 12:38:00,809 DEBUG lava-logs => PING(20) 2019-11-04 12:38:08,813 INFO Dhanu => HELLO 2019-11-04 12:38:08,814 WARNING Dispatcher has RESTARTED 2019-11-04 12:38:11,907 INFO [POLL] Received a signal, leaving 2019-11-04 12:38:11,908 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:38:12,912 INFO [INIT] Dropping privileges 2019-11-04 12:38:12,912 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:38:12,913 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:38:12,914 INFO [INIT] Marking all workers as offline 2019-11-04 12:38:12,945 INFO [INIT] Starting encryption 2019-11-04 12:38:12,946 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:38:12,947 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:38:12,947 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:38:12,960 INFO [INIT] LAVA master has started. 2019-11-04 12:38:12,960 INFO [INIT] Using protocol version 3 2019-11-04 12:38:13,816 DEBUG Dhanu => PING(20) 2019-11-04 12:38:13,816 WARNING Unknown dispatcher (server crashed) 2019-11-04 12:38:20,826 DEBUG lava-logs => PING(20) 2019-11-04 12:38:32,972 INFO scheduling health checks: 2019-11-04 12:38:32,979 INFO scheduling jobs: 2019-11-04 12:38:33,837 DEBUG Dhanu => PING(20) 2019-11-04 12:38:40,846 DEBUG lava-logs => PING(20) 2019-11-04 12:38:52,986 INFO scheduling health checks: 2019-11-04 12:38:52,996 INFO scheduling jobs: 2019-11-04 12:38:53,858 DEBUG Dhanu => PING(20) 2019-11-04 12:39:00,869 DEBUG lava-logs => PING(20) 2019-11-04 12:39:13,012 INFO scheduling health checks: 2019-11-04 12:39:13,026 INFO scheduling jobs: 2019-11-04 12:39:13,878 DEBUG Dhanu => PING(20) 2019-11-04 12:39:20,892 DEBUG lava-logs => PING(20) 2019-11-04 12:39:33,038 INFO scheduling health checks: 2019-11-04 12:39:33,047 INFO scheduling jobs: 2019-11-04 12:39:33,903 DEBUG Dhanu => PING(20) 2019-11-04 12:39:40,905 DEBUG lava-logs => PING(20) 2019-11-04 12:39:53,060 INFO scheduling health checks: 2019-11-04 12:39:53,071 INFO scheduling jobs: 2019-11-04 12:39:53,928 DEBUG Dhanu => PING(20) 2019-11-04 12:40:00,927 DEBUG lava-logs => PING(20) 2019-11-04 12:40:13,080 INFO scheduling health checks: 2019-11-04 12:40:13,090 INFO scheduling jobs: 2019-11-04 12:40:13,953 DEBUG Dhanu => PING(20) 2019-11-04 12:40:20,941 DEBUG lava-logs => PING(20) 2019-11-04 12:40:33,102 INFO scheduling health checks: 2019-11-04 12:40:33,114 INFO scheduling jobs: 2019-11-04 12:40:33,972 DEBUG Dhanu => PING(20) 2019-11-04 12:40:40,963 DEBUG lava-logs => PING(20) 2019-11-04 12:40:53,127 INFO scheduling health checks: 2019-11-04 12:40:53,138 INFO scheduling jobs: 2019-11-04 12:40:53,991 DEBUG Dhanu => PING(20) 2019-11-04 12:41:00,982 DEBUG lava-logs => PING(20) 2019-11-04 12:41:13,152 INFO scheduling health checks: 2019-11-04 12:41:13,160 INFO scheduling jobs: 2019-11-04 12:41:14,010 DEBUG Dhanu => PING(20) 2019-11-04 12:41:21,034 DEBUG lava-logs => PING(20) 2019-11-04 12:41:33,172 INFO scheduling health checks: 2019-11-04 12:41:33,182 INFO scheduling jobs: 2019-11-04 12:41:34,029 DEBUG Dhanu => PING(20) 2019-11-04 12:41:41,056 DEBUG lava-logs => PING(20) 2019-11-04 12:41:53,191 INFO scheduling health checks: 2019-11-04 12:41:53,201 INFO scheduling jobs: 2019-11-04 12:41:54,054 DEBUG Dhanu => PING(20) 2019-11-04 12:42:01,079 DEBUG lava-logs => PING(20) 2019-11-04 12:42:13,210 INFO scheduling health checks: 2019-11-04 12:42:13,221 INFO scheduling jobs: 2019-11-04 12:42:14,070 DEBUG Dhanu => PING(20) 2019-11-04 12:42:21,102 DEBUG lava-logs => PING(20) 2019-11-04 12:42:33,235 INFO scheduling health checks: 2019-11-04 12:42:33,246 INFO scheduling jobs: 2019-11-04 12:42:34,094 DEBUG Dhanu => PING(20) 2019-11-04 12:42:41,122 DEBUG lava-logs => PING(20) 2019-11-04 12:42:53,256 INFO scheduling health checks: 2019-11-04 12:42:53,267 INFO scheduling jobs: 2019-11-04 12:42:54,114 DEBUG Dhanu => PING(20) 2019-11-04 12:43:01,141 DEBUG lava-logs => PING(20) 2019-11-04 12:43:13,279 INFO scheduling health checks: 2019-11-04 12:43:13,286 INFO scheduling jobs: 2019-11-04 12:43:14,135 DEBUG Dhanu => PING(20) 2019-11-04 12:43:21,164 DEBUG lava-logs => PING(20) 2019-11-04 12:43:33,297 INFO scheduling health checks: 2019-11-04 12:43:33,309 INFO scheduling jobs: 2019-11-04 12:43:34,157 DEBUG Dhanu => PING(20) 2019-11-04 12:43:37,078 DEBUG lava-logs => PING(20) 2019-11-04 12:43:37,098 DEBUG lava-logs => PING(20) 2019-11-04 12:43:53,318 INFO scheduling health checks: 2019-11-04 12:43:53,329 INFO scheduling jobs: 2019-11-04 12:43:54,179 DEBUG Dhanu => PING(20) 2019-11-04 12:43:57,119 DEBUG lava-logs => PING(20) 2019-11-04 12:44:01,333 INFO [POLL] Received a signal, leaving 2019-11-04 12:44:01,333 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 12:44:02,317 INFO [INIT] Dropping privileges 2019-11-04 12:44:02,317 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 12:44:02,318 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 12:44:02,319 INFO [INIT] Marking all workers as offline 2019-11-04 12:44:02,345 INFO [INIT] Starting encryption 2019-11-04 12:44:02,348 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 12:44:02,349 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:44:02,349 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 12:44:02,363 INFO [INIT] LAVA master has started. 2019-11-04 12:44:02,363 INFO [INIT] Using protocol version 3 2019-11-04 12:44:06,588 INFO Dhanu => HELLO 2019-11-04 12:44:06,588 WARNING New dispatcher 2019-11-04 12:44:11,592 DEBUG Dhanu => PING(20) 2019-11-04 12:44:17,134 DEBUG lava-logs => PING(20) 2019-11-04 12:44:22,369 INFO scheduling health checks: 2019-11-04 12:44:22,383 INFO scheduling jobs: 2019-11-04 12:44:31,614 DEBUG Dhanu => PING(20) 2019-11-04 12:44:37,154 DEBUG lava-logs => PING(20) 2019-11-04 12:44:42,392 INFO scheduling health checks: 2019-11-04 12:44:42,404 INFO scheduling jobs: 2019-11-04 12:44:51,638 DEBUG Dhanu => PING(20) 2019-11-04 12:44:57,173 DEBUG lava-logs => PING(20) 2019-11-04 12:45:02,414 INFO scheduling health checks: 2019-11-04 12:45:02,425 INFO scheduling jobs: 2019-11-04 12:45:11,663 DEBUG Dhanu => PING(20) 2019-11-04 12:45:17,195 DEBUG lava-logs => PING(20) 2019-11-04 12:45:22,436 INFO scheduling health checks: 2019-11-04 12:45:22,448 INFO scheduling jobs: 2019-11-04 12:45:31,688 DEBUG Dhanu => PING(20) 2019-11-04 12:45:37,218 DEBUG lava-logs => PING(20) 2019-11-04 12:45:42,458 INFO scheduling health checks: 2019-11-04 12:45:42,468 INFO scheduling jobs: 2019-11-04 12:45:51,712 DEBUG Dhanu => PING(20) 2019-11-04 12:45:57,241 DEBUG lava-logs => PING(20) 2019-11-04 12:46:02,479 INFO scheduling health checks: 2019-11-04 12:46:02,490 INFO scheduling jobs: 2019-11-04 12:46:11,734 DEBUG Dhanu => PING(20) 2019-11-04 12:46:17,264 DEBUG lava-logs => PING(20) 2019-11-04 12:46:22,501 INFO scheduling health checks: 2019-11-04 12:46:22,511 INFO scheduling jobs: 2019-11-04 12:46:31,754 DEBUG Dhanu => PING(20) 2019-11-04 12:46:37,287 DEBUG lava-logs => PING(20) 2019-11-04 12:46:42,523 INFO scheduling health checks: 2019-11-04 12:46:42,533 INFO scheduling jobs: 2019-11-04 12:46:51,778 DEBUG Dhanu => PING(20) 2019-11-04 12:46:57,309 DEBUG lava-logs => PING(20) 2019-11-04 12:47:02,545 INFO scheduling health checks: 2019-11-04 12:47:02,557 INFO scheduling jobs: 2019-11-04 12:47:11,801 DEBUG Dhanu => PING(20) 2019-11-04 12:47:17,332 DEBUG lava-logs => PING(20) 2019-11-04 12:47:22,567 INFO scheduling health checks: 2019-11-04 12:47:22,578 INFO scheduling jobs: 2019-11-04 12:47:31,822 DEBUG Dhanu => PING(20) 2019-11-04 12:47:37,355 DEBUG lava-logs => PING(20) 2019-11-04 12:47:42,587 INFO scheduling health checks: 2019-11-04 12:47:42,599 INFO scheduling jobs: 2019-11-04 12:47:51,846 DEBUG Dhanu => PING(20) 2019-11-04 12:47:57,378 DEBUG lava-logs => PING(20) 2019-11-04 12:48:02,613 INFO scheduling health checks: 2019-11-04 12:48:02,618 INFO scheduling jobs: 2019-11-04 12:48:11,863 DEBUG Dhanu => PING(20) 2019-11-04 12:48:17,397 DEBUG lava-logs => PING(20) 2019-11-04 12:48:22,630 INFO scheduling health checks: 2019-11-04 12:48:22,641 INFO scheduling jobs: 2019-11-04 12:48:31,886 DEBUG Dhanu => PING(20) 2019-11-04 12:48:37,413 DEBUG lava-logs => PING(20) 2019-11-04 12:48:42,653 INFO scheduling health checks: 2019-11-04 12:48:42,664 INFO scheduling jobs: 2019-11-04 12:48:51,911 DEBUG Dhanu => PING(20) 2019-11-04 12:48:57,436 DEBUG lava-logs => PING(20) 2019-11-04 12:49:02,674 INFO scheduling health checks: 2019-11-04 12:49:02,684 INFO scheduling jobs: 2019-11-04 12:49:11,929 DEBUG Dhanu => PING(20) 2019-11-04 12:49:17,456 DEBUG lava-logs => PING(20) 2019-11-04 12:49:22,695 INFO scheduling health checks: 2019-11-04 12:49:22,705 INFO scheduling jobs: 2019-11-04 12:49:31,952 DEBUG Dhanu => PING(20) 2019-11-04 12:49:37,478 DEBUG lava-logs => PING(20) 2019-11-04 12:49:42,715 INFO scheduling health checks: 2019-11-04 12:49:42,725 INFO scheduling jobs: 2019-11-04 12:49:51,974 DEBUG Dhanu => PING(20) 2019-11-04 12:49:57,498 DEBUG lava-logs => PING(20) 2019-11-04 12:50:02,732 INFO scheduling health checks: 2019-11-04 12:50:02,742 INFO scheduling jobs: 2019-11-04 12:50:12,015 DEBUG Dhanu => PING(20) 2019-11-04 12:50:17,514 DEBUG lava-logs => PING(20) 2019-11-04 12:50:22,754 INFO scheduling health checks: 2019-11-04 12:50:22,782 INFO scheduling jobs: 2019-11-04 12:50:32,032 DEBUG Dhanu => PING(20) 2019-11-04 12:50:37,537 DEBUG lava-logs => PING(20) 2019-11-04 12:50:42,793 INFO scheduling health checks: 2019-11-04 12:50:42,805 INFO scheduling jobs: 2019-11-04 12:50:52,057 DEBUG Dhanu => PING(20) 2019-11-04 12:50:57,560 DEBUG lava-logs => PING(20) 2019-11-04 12:51:02,817 INFO scheduling health checks: 2019-11-04 12:51:02,827 INFO scheduling jobs: 2019-11-04 12:51:12,081 DEBUG Dhanu => PING(20) 2019-11-04 12:51:17,583 DEBUG lava-logs => PING(20) 2019-11-04 12:51:22,837 INFO scheduling health checks: 2019-11-04 12:51:22,847 INFO scheduling jobs: 2019-11-04 12:51:32,102 DEBUG Dhanu => PING(20) 2019-11-04 12:51:37,605 DEBUG lava-logs => PING(20) 2019-11-04 12:51:42,862 INFO scheduling health checks: 2019-11-04 12:51:42,873 INFO scheduling jobs: 2019-11-04 12:51:52,123 DEBUG Dhanu => PING(20) 2019-11-04 12:51:57,624 DEBUG lava-logs => PING(20) 2019-11-04 12:52:02,888 INFO scheduling health checks: 2019-11-04 12:52:02,897 INFO scheduling jobs: 2019-11-04 12:52:12,147 DEBUG Dhanu => PING(20) 2019-11-04 12:52:17,634 DEBUG lava-logs => PING(20) 2019-11-04 12:52:22,905 INFO scheduling health checks: 2019-11-04 12:52:22,915 INFO scheduling jobs: 2019-11-04 12:52:32,170 DEBUG Dhanu => PING(20) 2019-11-04 12:52:37,657 DEBUG lava-logs => PING(20) 2019-11-04 12:52:42,923 INFO scheduling health checks: 2019-11-04 12:52:42,931 INFO scheduling jobs: 2019-11-04 12:52:52,194 DEBUG Dhanu => PING(20) 2019-11-04 12:52:57,680 DEBUG lava-logs => PING(20) 2019-11-04 12:53:02,941 INFO scheduling health checks: 2019-11-04 12:53:02,950 INFO scheduling jobs: 2019-11-04 12:53:12,213 DEBUG Dhanu => PING(20) 2019-11-04 12:53:17,701 DEBUG lava-logs => PING(20) 2019-11-04 12:53:22,959 INFO scheduling health checks: 2019-11-04 12:53:22,970 INFO scheduling jobs: 2019-11-04 12:53:32,235 DEBUG Dhanu => PING(20) 2019-11-04 12:53:37,724 DEBUG lava-logs => PING(20) 2019-11-04 12:53:42,980 INFO scheduling health checks: 2019-11-04 12:53:42,990 INFO scheduling jobs: 2019-11-04 12:53:52,260 DEBUG Dhanu => PING(20) 2019-11-04 12:53:57,742 DEBUG lava-logs => PING(20) 2019-11-04 12:54:03,002 INFO scheduling health checks: 2019-11-04 12:54:03,013 INFO scheduling jobs: 2019-11-04 12:54:12,284 DEBUG Dhanu => PING(20) 2019-11-04 12:54:17,760 DEBUG lava-logs => PING(20) 2019-11-04 12:54:23,022 INFO scheduling health checks: 2019-11-04 12:54:23,054 INFO scheduling jobs: 2019-11-04 12:54:32,368 DEBUG Dhanu => PING(20) 2019-11-04 12:54:37,783 DEBUG lava-logs => PING(20) 2019-11-04 12:54:43,067 INFO scheduling health checks: 2019-11-04 12:54:43,076 INFO scheduling jobs: 2019-11-04 12:54:52,405 DEBUG Dhanu => PING(20) 2019-11-04 12:54:57,800 DEBUG lava-logs => PING(20) 2019-11-04 12:55:03,088 INFO scheduling health checks: 2019-11-04 12:55:03,093 INFO scheduling jobs: 2019-11-04 12:55:12,425 DEBUG Dhanu => PING(20) 2019-11-04 12:55:17,817 DEBUG lava-logs => PING(20) 2019-11-04 12:55:23,105 INFO scheduling health checks: 2019-11-04 12:55:23,116 INFO scheduling jobs: 2019-11-04 12:55:32,450 DEBUG Dhanu => PING(20) 2019-11-04 12:55:37,840 DEBUG lava-logs => PING(20) 2019-11-04 12:55:43,142 INFO scheduling health checks: 2019-11-04 12:55:43,184 INFO scheduling jobs: 2019-11-04 12:55:52,629 DEBUG Dhanu => PING(20) 2019-11-04 12:55:57,863 DEBUG lava-logs => PING(20) 2019-11-04 12:56:03,194 INFO scheduling health checks: 2019-11-04 12:56:03,206 INFO scheduling jobs: 2019-11-04 12:56:12,649 DEBUG Dhanu => PING(20) 2019-11-04 12:56:17,886 DEBUG lava-logs => PING(20) 2019-11-04 12:56:23,216 INFO scheduling health checks: 2019-11-04 12:56:23,226 INFO scheduling jobs: 2019-11-04 12:56:32,669 DEBUG Dhanu => PING(20) 2019-11-04 12:56:37,904 DEBUG lava-logs => PING(20) 2019-11-04 12:56:43,238 INFO scheduling health checks: 2019-11-04 12:56:43,285 INFO scheduling jobs: 2019-11-04 12:56:52,712 DEBUG Dhanu => PING(20) 2019-11-04 12:56:57,927 DEBUG lava-logs => PING(20) 2019-11-04 12:57:03,297 INFO scheduling health checks: 2019-11-04 12:57:03,309 INFO scheduling jobs: 2019-11-04 12:57:12,733 DEBUG Dhanu => PING(20) 2019-11-04 12:57:17,950 DEBUG lava-logs => PING(20) 2019-11-04 12:57:23,318 INFO scheduling health checks: 2019-11-04 12:57:23,327 INFO scheduling jobs: 2019-11-04 12:57:32,757 DEBUG Dhanu => PING(20) 2019-11-04 12:57:37,968 DEBUG lava-logs => PING(20) 2019-11-04 12:57:43,340 INFO scheduling health checks: 2019-11-04 12:57:43,351 INFO scheduling jobs: 2019-11-04 12:57:52,777 DEBUG Dhanu => PING(20) 2019-11-04 12:57:57,991 DEBUG lava-logs => PING(20) 2019-11-04 12:58:03,362 INFO scheduling health checks: 2019-11-04 12:58:03,392 INFO scheduling jobs: 2019-11-04 12:58:12,814 DEBUG Dhanu => PING(20) 2019-11-04 12:58:18,014 DEBUG lava-logs => PING(20) 2019-11-04 12:58:23,402 INFO scheduling health checks: 2019-11-04 12:58:23,408 INFO scheduling jobs: 2019-11-04 12:58:32,838 DEBUG Dhanu => PING(20) 2019-11-04 12:58:38,034 DEBUG lava-logs => PING(20) 2019-11-04 12:58:43,417 INFO scheduling health checks: 2019-11-04 12:58:43,428 INFO scheduling jobs: 2019-11-04 12:58:52,858 DEBUG Dhanu => PING(20) 2019-11-04 12:58:58,057 DEBUG lava-logs => PING(20) 2019-11-04 12:59:03,446 INFO scheduling health checks: 2019-11-04 12:59:03,457 INFO scheduling jobs: 2019-11-04 12:59:12,876 DEBUG Dhanu => PING(20) 2019-11-04 12:59:18,080 DEBUG lava-logs => PING(20) 2019-11-04 12:59:23,467 INFO scheduling health checks: 2019-11-04 12:59:23,535 INFO scheduling jobs: 2019-11-04 12:59:32,898 DEBUG Dhanu => PING(20) 2019-11-04 12:59:38,099 DEBUG lava-logs => PING(20) 2019-11-04 12:59:43,547 INFO scheduling health checks: 2019-11-04 12:59:43,559 INFO scheduling jobs: 2019-11-04 12:59:52,922 DEBUG Dhanu => PING(20) 2019-11-04 12:59:58,122 DEBUG lava-logs => PING(20) 2019-11-04 13:00:03,570 INFO scheduling health checks: 2019-11-04 13:00:03,581 INFO scheduling jobs: 2019-11-04 13:00:12,943 DEBUG Dhanu => PING(20) 2019-11-04 13:00:18,145 DEBUG lava-logs => PING(20) 2019-11-04 13:00:23,591 INFO scheduling health checks: 2019-11-04 13:00:23,603 INFO scheduling jobs: 2019-11-04 13:00:32,959 DEBUG Dhanu => PING(20) 2019-11-04 13:00:38,182 DEBUG lava-logs => PING(20) 2019-11-04 13:00:43,613 INFO scheduling health checks: 2019-11-04 13:00:43,623 INFO scheduling jobs: 2019-11-04 13:00:52,982 DEBUG Dhanu => PING(20) 2019-11-04 13:00:58,205 DEBUG lava-logs => PING(20) 2019-11-04 13:01:03,638 INFO scheduling health checks: 2019-11-04 13:01:03,649 INFO scheduling jobs: 2019-11-04 13:01:13,002 DEBUG Dhanu => PING(20) 2019-11-04 13:01:18,225 DEBUG lava-logs => PING(20) 2019-11-04 13:01:23,662 INFO scheduling health checks: 2019-11-04 13:01:23,672 INFO scheduling jobs: 2019-11-04 13:01:33,023 DEBUG Dhanu => PING(20) 2019-11-04 13:01:38,248 DEBUG lava-logs => PING(20) 2019-11-04 13:01:43,683 INFO scheduling health checks: 2019-11-04 13:01:43,695 INFO scheduling jobs: 2019-11-04 13:01:53,048 DEBUG Dhanu => PING(20) 2019-11-04 13:01:58,262 DEBUG lava-logs => PING(20) 2019-11-04 13:02:03,710 INFO scheduling health checks: 2019-11-04 13:02:03,722 INFO scheduling jobs: 2019-11-04 13:02:13,070 DEBUG Dhanu => PING(20) 2019-11-04 13:02:18,285 DEBUG lava-logs => PING(20) 2019-11-04 13:02:23,734 INFO scheduling health checks: 2019-11-04 13:02:23,745 INFO scheduling jobs: 2019-11-04 13:02:33,094 DEBUG Dhanu => PING(20) 2019-11-04 13:02:38,308 DEBUG lava-logs => PING(20) 2019-11-04 13:02:43,756 INFO scheduling health checks: 2019-11-04 13:02:43,767 INFO scheduling jobs: 2019-11-04 13:02:53,117 DEBUG Dhanu => PING(20) 2019-11-04 13:02:58,331 DEBUG lava-logs => PING(20) 2019-11-04 13:03:03,780 INFO scheduling health checks: 2019-11-04 13:03:03,790 INFO scheduling jobs: 2019-11-04 13:03:13,140 DEBUG Dhanu => PING(20) 2019-11-04 13:03:18,351 DEBUG lava-logs => PING(20) 2019-11-04 13:03:23,803 INFO scheduling health checks: 2019-11-04 13:03:23,810 INFO scheduling jobs: 2019-11-04 13:03:33,165 DEBUG Dhanu => PING(20) 2019-11-04 13:03:38,365 DEBUG lava-logs => PING(20) 2019-11-04 13:03:43,823 INFO scheduling health checks: 2019-11-04 13:03:43,835 INFO scheduling jobs: 2019-11-04 13:03:53,189 DEBUG Dhanu => PING(20) 2019-11-04 13:03:58,388 DEBUG lava-logs => PING(20) 2019-11-04 13:04:03,846 INFO scheduling health checks: 2019-11-04 13:04:03,857 INFO scheduling jobs: 2019-11-04 13:12:20,627 DEBUG Dhanu => PING(20) 2019-11-04 13:12:20,743 DEBUG lava-logs => PING(20) 2019-11-04 13:12:20,845 INFO scheduling health checks: 2019-11-04 13:12:20,854 INFO scheduling jobs: 2019-11-04 13:12:40,649 DEBUG Dhanu => PING(20) 2019-11-04 13:12:40,761 DEBUG lava-logs => PING(20) 2019-11-04 13:12:40,864 INFO scheduling health checks: 2019-11-04 13:12:40,870 INFO scheduling jobs: 2019-11-04 13:13:00,667 DEBUG Dhanu => PING(20) 2019-11-04 13:13:00,779 DEBUG lava-logs => PING(20) 2019-11-04 13:13:00,880 INFO scheduling health checks: 2019-11-04 13:13:00,892 INFO scheduling jobs: 2019-11-04 13:13:20,689 DEBUG Dhanu => PING(20) 2019-11-04 13:13:20,798 DEBUG lava-logs => PING(20) 2019-11-04 13:13:20,909 INFO scheduling health checks: 2019-11-04 13:13:20,919 INFO scheduling jobs: 2019-11-04 13:13:40,709 DEBUG Dhanu => PING(20) 2019-11-04 13:13:40,817 DEBUG lava-logs => PING(20) 2019-11-04 13:13:40,929 INFO scheduling health checks: 2019-11-04 13:13:40,939 INFO scheduling jobs: 2019-11-04 13:14:00,726 DEBUG Dhanu => PING(20) 2019-11-04 13:14:00,837 DEBUG lava-logs => PING(20) 2019-11-04 13:14:00,949 INFO scheduling health checks: 2019-11-04 13:14:00,956 INFO scheduling jobs: 2019-11-04 13:14:20,742 DEBUG Dhanu => PING(20) 2019-11-04 13:14:20,857 DEBUG lava-logs => PING(20) 2019-11-04 13:14:20,967 INFO scheduling health checks: 2019-11-04 13:14:20,979 INFO scheduling jobs: 2019-11-04 13:14:40,757 DEBUG Dhanu => PING(20) 2019-11-04 13:14:40,880 DEBUG lava-logs => PING(20) 2019-11-04 13:14:40,988 INFO scheduling health checks: 2019-11-04 13:14:40,999 INFO scheduling jobs: 2019-11-04 13:15:00,777 DEBUG Dhanu => PING(20) 2019-11-04 13:15:00,902 DEBUG lava-logs => PING(20) 2019-11-04 13:15:01,010 INFO scheduling health checks: 2019-11-04 13:15:01,020 INFO scheduling jobs: 2019-11-04 13:15:20,801 DEBUG Dhanu => PING(20) 2019-11-04 13:15:20,925 DEBUG lava-logs => PING(20) 2019-11-04 13:15:21,033 INFO scheduling health checks: 2019-11-04 13:15:21,043 INFO scheduling jobs: 2019-11-04 13:15:40,818 DEBUG Dhanu => PING(20) 2019-11-04 13:15:40,944 DEBUG lava-logs => PING(20) 2019-11-04 13:15:41,053 INFO scheduling health checks: 2019-11-04 13:15:41,064 INFO scheduling jobs: 2019-11-04 13:16:00,839 DEBUG Dhanu => PING(20) 2019-11-04 13:16:00,967 DEBUG lava-logs => PING(20) 2019-11-04 13:16:01,075 INFO scheduling health checks: 2019-11-04 13:16:01,085 INFO scheduling jobs: 2019-11-04 13:16:20,863 DEBUG Dhanu => PING(20) 2019-11-04 13:16:21,094 INFO scheduling health checks: 2019-11-04 13:16:21,105 INFO scheduling jobs: 2019-11-04 13:16:22,856 DEBUG lava-logs => PING(20) 2019-11-04 13:16:22,912 DEBUG lava-logs => PING(20) 2019-11-04 13:16:26,546 INFO Dhanu => HELLO 2019-11-04 13:16:26,547 WARNING Dispatcher has RESTARTED 2019-11-04 13:16:29,223 INFO [POLL] Received a signal, leaving 2019-11-04 13:16:29,235 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:16:31,279 INFO [INIT] Dropping privileges 2019-11-04 13:16:31,279 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:16:31,279 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:16:31,281 INFO [INIT] Marking all workers as offline 2019-11-04 13:16:31,351 INFO [INIT] Starting encryption 2019-11-04 13:16:31,354 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:16:31,354 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:16:31,355 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:16:31,372 INFO [INIT] LAVA master has started. 2019-11-04 13:16:31,372 INFO [INIT] Using protocol version 3 2019-11-04 13:16:31,549 DEBUG Dhanu => PING(20) 2019-11-04 13:16:31,549 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:16:42,926 DEBUG lava-logs => PING(20) 2019-11-04 13:16:51,381 INFO scheduling health checks: 2019-11-04 13:16:51,398 INFO scheduling jobs: 2019-11-04 13:16:51,565 DEBUG Dhanu => PING(20) 2019-11-04 13:17:02,945 DEBUG lava-logs => PING(20) 2019-11-04 13:17:11,408 INFO scheduling health checks: 2019-11-04 13:17:11,421 INFO scheduling jobs: 2019-11-04 13:17:11,581 DEBUG Dhanu => PING(20) 2019-11-04 13:17:22,962 DEBUG lava-logs => PING(20) 2019-11-04 13:17:31,433 INFO scheduling health checks: 2019-11-04 13:17:31,436 INFO scheduling jobs: 2019-11-04 13:17:31,597 DEBUG Dhanu => PING(20) 2019-11-04 13:17:42,981 DEBUG lava-logs => PING(20) 2019-11-04 13:17:51,453 INFO scheduling health checks: 2019-11-04 13:17:51,457 INFO scheduling jobs: 2019-11-04 13:17:51,621 DEBUG Dhanu => PING(20) 2019-11-04 13:18:02,994 DEBUG lava-logs => PING(20) 2019-11-04 13:18:11,470 INFO scheduling health checks: 2019-11-04 13:18:11,474 INFO scheduling jobs: 2019-11-04 13:18:11,645 DEBUG Dhanu => PING(20) 2019-11-04 13:18:28,048 DEBUG lava-logs => PING(20) 2019-11-04 13:18:28,068 DEBUG lava-logs => PING(20) 2019-11-04 13:18:31,492 INFO scheduling health checks: 2019-11-04 13:18:31,496 INFO scheduling jobs: 2019-11-04 13:18:31,670 DEBUG Dhanu => PING(20) 2019-11-04 13:18:48,091 DEBUG lava-logs => PING(20) 2019-11-04 13:18:49,188 INFO Dhanu => HELLO 2019-11-04 13:18:49,188 WARNING Dispatcher has RESTARTED 2019-11-04 13:18:51,514 INFO scheduling health checks: 2019-11-04 13:18:51,518 INFO scheduling jobs: 2019-11-04 13:18:53,277 INFO [POLL] Received a signal, leaving 2019-11-04 13:18:53,277 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:18:54,254 INFO [INIT] Dropping privileges 2019-11-04 13:18:54,254 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:18:54,255 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:18:54,257 INFO [INIT] Marking all workers as offline 2019-11-04 13:18:54,327 INFO [INIT] Starting encryption 2019-11-04 13:18:54,329 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:18:54,329 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:18:54,330 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:18:54,344 INFO [INIT] LAVA master has started. 2019-11-04 13:18:54,344 INFO [INIT] Using protocol version 3 2019-11-04 13:18:54,487 DEBUG Dhanu => PING(20) 2019-11-04 13:18:54,488 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:19:08,108 DEBUG lava-logs => PING(20) 2019-11-04 13:19:14,350 INFO scheduling health checks: 2019-11-04 13:19:14,356 INFO scheduling jobs: 2019-11-04 13:19:14,509 DEBUG Dhanu => PING(20) 2019-11-04 13:19:28,130 DEBUG lava-logs => PING(20) 2019-11-04 13:19:34,370 INFO scheduling health checks: 2019-11-04 13:19:34,373 INFO scheduling jobs: 2019-11-04 13:19:34,532 DEBUG Dhanu => PING(20) 2019-11-04 13:19:48,153 DEBUG lava-logs => PING(20) 2019-11-04 13:19:54,386 INFO scheduling health checks: 2019-11-04 13:19:54,390 INFO scheduling jobs: 2019-11-04 13:19:54,552 DEBUG Dhanu => PING(20) 2019-11-04 13:20:08,176 DEBUG lava-logs => PING(20) 2019-11-04 13:20:14,405 INFO scheduling health checks: 2019-11-04 13:20:14,409 INFO scheduling jobs: 2019-11-04 13:20:14,577 DEBUG Dhanu => PING(20) 2019-11-04 13:20:28,190 DEBUG lava-logs => PING(20) 2019-11-04 13:20:34,423 INFO scheduling health checks: 2019-11-04 13:20:34,427 INFO scheduling jobs: 2019-11-04 13:20:34,601 DEBUG Dhanu => PING(20) 2019-11-04 13:20:48,212 DEBUG lava-logs => PING(20) 2019-11-04 13:20:54,441 INFO scheduling health checks: 2019-11-04 13:20:54,444 INFO scheduling jobs: 2019-11-04 13:20:54,623 DEBUG Dhanu => PING(20) 2019-11-04 13:21:08,233 DEBUG lava-logs => PING(20) 2019-11-04 13:21:14,458 INFO scheduling health checks: 2019-11-04 13:21:14,462 INFO scheduling jobs: 2019-11-04 13:21:14,643 DEBUG Dhanu => PING(20) 2019-11-04 13:21:28,358 DEBUG lava-logs => PING(20) 2019-11-04 13:21:34,476 INFO scheduling health checks: 2019-11-04 13:21:34,517 INFO scheduling jobs: 2019-11-04 13:21:34,676 DEBUG Dhanu => PING(20) 2019-11-04 13:21:48,400 DEBUG lava-logs => PING(20) 2019-11-04 13:21:54,533 INFO scheduling health checks: 2019-11-04 13:21:54,537 INFO scheduling jobs: 2019-11-04 13:21:54,701 DEBUG Dhanu => PING(20) 2019-11-04 13:22:08,423 DEBUG lava-logs => PING(20) 2019-11-04 13:22:14,552 INFO scheduling health checks: 2019-11-04 13:22:14,556 INFO scheduling jobs: 2019-11-04 13:22:14,725 DEBUG Dhanu => PING(20) 2019-11-04 13:22:28,446 DEBUG lava-logs => PING(20) 2019-11-04 13:22:34,574 INFO scheduling health checks: 2019-11-04 13:22:34,578 INFO scheduling jobs: 2019-11-04 13:22:34,746 DEBUG Dhanu => PING(20) 2019-11-04 13:22:48,468 DEBUG lava-logs => PING(20) 2019-11-04 13:22:54,594 INFO scheduling health checks: 2019-11-04 13:22:54,597 INFO scheduling jobs: 2019-11-04 13:22:54,766 DEBUG Dhanu => PING(20) 2019-11-04 13:23:08,486 DEBUG lava-logs => PING(20) 2019-11-04 13:23:14,611 INFO scheduling health checks: 2019-11-04 13:23:14,615 INFO scheduling jobs: 2019-11-04 13:23:14,790 DEBUG Dhanu => PING(20) 2019-11-04 13:23:28,509 DEBUG lava-logs => PING(20) 2019-11-04 13:23:34,628 INFO scheduling health checks: 2019-11-04 13:23:34,632 INFO scheduling jobs: 2019-11-04 13:23:34,809 DEBUG Dhanu => PING(20) 2019-11-04 13:23:47,153 INFO [POLL] Received a signal, leaving 2019-11-04 13:23:47,153 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:23:49,769 INFO [INIT] Dropping privileges 2019-11-04 13:23:49,770 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:23:49,770 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:23:49,771 INFO [INIT] Marking all workers as offline 2019-11-04 13:23:49,811 INFO [INIT] Starting encryption 2019-11-04 13:23:49,813 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:23:49,814 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:23:49,814 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:23:49,833 INFO [INIT] LAVA master has started. 2019-11-04 13:23:49,833 INFO [INIT] Using protocol version 3 2019-11-04 13:23:49,843 DEBUG lava-logs => PING(20) 2019-11-04 13:23:54,829 DEBUG Dhanu => PING(20) 2019-11-04 13:23:54,830 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:24:09,837 INFO scheduling health checks: 2019-11-04 13:24:09,843 INFO scheduling jobs: 2019-11-04 13:24:09,861 DEBUG lava-logs => PING(20) 2019-11-04 13:24:14,850 DEBUG Dhanu => PING(20) 2019-11-04 13:24:29,859 INFO scheduling health checks: 2019-11-04 13:24:29,863 INFO scheduling jobs: 2019-11-04 13:24:29,883 DEBUG lava-logs => PING(20) 2019-11-04 13:24:34,873 DEBUG Dhanu => PING(20) 2019-11-04 13:24:49,877 INFO scheduling health checks: 2019-11-04 13:24:49,881 INFO scheduling jobs: 2019-11-04 13:24:49,901 DEBUG lava-logs => PING(20) 2019-11-04 13:24:54,898 DEBUG Dhanu => PING(20) 2019-11-04 13:25:09,894 INFO scheduling health checks: 2019-11-04 13:25:09,898 INFO scheduling jobs: 2019-11-04 13:25:09,913 DEBUG lava-logs => PING(20) 2019-11-04 13:25:14,921 DEBUG Dhanu => PING(20) 2019-11-04 13:25:29,912 INFO scheduling health checks: 2019-11-04 13:25:29,916 INFO scheduling jobs: 2019-11-04 13:25:29,934 DEBUG lava-logs => PING(20) 2019-11-04 13:25:34,941 DEBUG Dhanu => PING(20) 2019-11-04 13:25:49,935 INFO scheduling health checks: 2019-11-04 13:25:49,939 INFO scheduling jobs: 2019-11-04 13:25:49,953 DEBUG lava-logs => PING(20) 2019-11-04 13:25:54,960 DEBUG Dhanu => PING(20) 2019-11-04 13:26:09,954 INFO scheduling health checks: 2019-11-04 13:26:09,959 INFO scheduling jobs: 2019-11-04 13:26:09,973 DEBUG lava-logs => PING(20) 2019-11-04 13:26:14,983 DEBUG Dhanu => PING(20) 2019-11-04 13:26:29,974 INFO scheduling health checks: 2019-11-04 13:26:29,976 INFO scheduling jobs: 2019-11-04 13:26:29,992 DEBUG lava-logs => PING(20) 2019-11-04 13:26:35,003 DEBUG Dhanu => PING(20) 2019-11-04 13:26:49,993 INFO scheduling health checks: 2019-11-04 13:26:50,005 INFO scheduling jobs: 2019-11-04 13:26:50,020 DEBUG lava-logs => PING(20) 2019-11-04 13:26:55,023 DEBUG Dhanu => PING(20) 2019-11-04 13:27:10,020 INFO scheduling health checks: 2019-11-04 13:27:10,030 INFO scheduling jobs: 2019-11-04 13:27:10,039 DEBUG lava-logs => PING(20) 2019-11-04 13:27:15,047 DEBUG Dhanu => PING(20) 2019-11-04 13:27:30,041 INFO scheduling health checks: 2019-11-04 13:27:30,051 INFO scheduling jobs: 2019-11-04 13:27:30,061 DEBUG lava-logs => PING(20) 2019-11-04 13:27:35,064 DEBUG Dhanu => PING(20) 2019-11-04 13:27:50,062 INFO scheduling health checks: 2019-11-04 13:27:50,072 INFO scheduling jobs: 2019-11-04 13:27:50,085 DEBUG lava-logs => PING(20) 2019-11-04 13:27:55,083 DEBUG Dhanu => PING(20) 2019-11-04 13:28:10,085 INFO scheduling health checks: 2019-11-04 13:28:10,096 INFO scheduling jobs: 2019-11-04 13:28:10,110 DEBUG lava-logs => PING(20) 2019-11-04 13:28:15,104 DEBUG Dhanu => PING(20) 2019-11-04 13:28:30,111 INFO scheduling health checks: 2019-11-04 13:28:30,122 INFO scheduling jobs: 2019-11-04 13:28:30,137 DEBUG lava-logs => PING(20) 2019-11-04 13:28:35,123 DEBUG Dhanu => PING(20) 2019-11-04 13:28:50,138 INFO scheduling health checks: 2019-11-04 13:28:50,149 INFO scheduling jobs: 2019-11-04 13:28:50,160 DEBUG lava-logs => PING(20) 2019-11-04 13:28:55,142 DEBUG Dhanu => PING(20) 2019-11-04 13:29:10,160 INFO scheduling health checks: 2019-11-04 13:29:10,197 INFO scheduling jobs: 2019-11-04 13:29:10,201 DEBUG - qemu 2019-11-04 13:29:10,229 DEBUG lava-logs => PING(20) 2019-11-04 13:29:15,165 DEBUG Dhanu => PING(20) 2019-11-04 13:29:30,230 INFO scheduling health checks: 2019-11-04 13:29:30,255 INFO scheduling jobs: 2019-11-04 13:29:30,258 DEBUG - qemu 2019-11-04 13:29:30,290 DEBUG lava-logs => PING(20) 2019-11-04 13:29:35,189 DEBUG Dhanu => PING(20) 2019-11-04 13:29:50,291 INFO scheduling health checks: 2019-11-04 13:29:50,335 INFO scheduling jobs: 2019-11-04 13:29:50,338 DEBUG - qemu 2019-11-04 13:29:50,369 DEBUG lava-logs => PING(20) 2019-11-04 13:29:55,214 DEBUG Dhanu => PING(20) 2019-11-04 13:30:10,370 INFO scheduling health checks: 2019-11-04 13:30:10,425 INFO scheduling jobs: 2019-11-04 13:30:10,426 DEBUG - qemu 2019-11-04 13:30:10,443 DEBUG lava-logs => PING(20) 2019-11-04 13:30:15,235 DEBUG Dhanu => PING(20) 2019-11-04 13:30:30,443 INFO scheduling health checks: 2019-11-04 13:30:30,472 INFO scheduling jobs: 2019-11-04 13:30:30,473 DEBUG - qemu 2019-11-04 13:30:30,589 DEBUG lava-logs => PING(20) 2019-11-04 13:30:35,259 DEBUG Dhanu => PING(20) 2019-11-04 13:30:50,589 INFO scheduling health checks: 2019-11-04 13:30:50,618 INFO scheduling jobs: 2019-11-04 13:30:50,622 DEBUG - qemu 2019-11-04 13:30:50,654 DEBUG lava-logs => PING(20) 2019-11-04 13:30:55,283 DEBUG Dhanu => PING(20) 2019-11-04 13:31:10,655 INFO scheduling health checks: 2019-11-04 13:31:10,687 INFO scheduling jobs: 2019-11-04 13:31:10,689 DEBUG - qemu 2019-11-04 13:31:10,721 DEBUG lava-logs => PING(20) 2019-11-04 13:31:15,303 DEBUG Dhanu => PING(20) 2019-11-04 13:31:30,722 INFO scheduling health checks: 2019-11-04 13:31:30,744 INFO scheduling jobs: 2019-11-04 13:31:30,746 DEBUG - qemu 2019-11-04 13:31:30,779 DEBUG lava-logs => PING(20) 2019-11-04 13:31:35,328 DEBUG Dhanu => PING(20) 2019-11-04 13:31:50,780 INFO scheduling health checks: 2019-11-04 13:31:50,813 INFO scheduling jobs: 2019-11-04 13:31:50,816 DEBUG - qemu 2019-11-04 13:31:50,843 DEBUG lava-logs => PING(20) 2019-11-04 13:31:55,350 DEBUG Dhanu => PING(20) 2019-11-04 13:32:10,844 INFO scheduling health checks: 2019-11-04 13:32:10,871 INFO scheduling jobs: 2019-11-04 13:32:10,873 DEBUG - qemu 2019-11-04 13:32:10,904 DEBUG lava-logs => PING(20) 2019-11-04 13:32:15,373 DEBUG Dhanu => PING(20) 2019-11-04 13:32:30,904 INFO scheduling health checks: 2019-11-04 13:32:30,928 INFO scheduling jobs: 2019-11-04 13:32:30,930 DEBUG - qemu 2019-11-04 13:32:30,962 DEBUG lava-logs => PING(20) 2019-11-04 13:32:35,397 DEBUG Dhanu => PING(20) 2019-11-04 13:32:50,963 INFO scheduling health checks: 2019-11-04 13:32:50,997 INFO scheduling jobs: 2019-11-04 13:32:51,000 DEBUG - qemu 2019-11-04 13:32:51,031 DEBUG lava-logs => PING(20) 2019-11-04 13:32:55,418 DEBUG Dhanu => PING(20) 2019-11-04 13:33:11,032 INFO scheduling health checks: 2019-11-04 13:33:11,065 INFO scheduling jobs: 2019-11-04 13:33:11,068 DEBUG - qemu 2019-11-04 13:33:11,098 DEBUG lava-logs => PING(20) 2019-11-04 13:33:15,442 DEBUG Dhanu => PING(20) 2019-11-04 13:33:31,099 INFO scheduling health checks: 2019-11-04 13:33:31,123 INFO scheduling jobs: 2019-11-04 13:33:31,125 DEBUG - qemu 2019-11-04 13:33:31,159 DEBUG lava-logs => PING(20) 2019-11-04 13:33:35,465 DEBUG Dhanu => PING(20) 2019-11-04 13:33:51,159 INFO scheduling health checks: 2019-11-04 13:33:51,192 INFO scheduling jobs: 2019-11-04 13:33:51,195 DEBUG - qemu 2019-11-04 13:33:51,227 DEBUG lava-logs => PING(20) 2019-11-04 13:33:55,489 DEBUG Dhanu => PING(20) 2019-11-04 13:34:11,228 INFO scheduling health checks: 2019-11-04 13:34:11,260 INFO scheduling jobs: 2019-11-04 13:34:11,263 DEBUG - qemu 2019-11-04 13:34:11,295 DEBUG lava-logs => PING(20) 2019-11-04 13:34:15,514 DEBUG Dhanu => PING(20) 2019-11-04 13:34:31,296 INFO scheduling health checks: 2019-11-04 13:34:31,329 INFO scheduling jobs: 2019-11-04 13:34:31,330 DEBUG - qemu 2019-11-04 13:34:31,349 DEBUG lava-logs => PING(20) 2019-11-04 13:34:35,533 DEBUG Dhanu => PING(20) 2019-11-04 13:34:51,350 INFO scheduling health checks: 2019-11-04 13:34:51,364 INFO scheduling jobs: 2019-11-04 13:34:51,366 DEBUG - qemu 2019-11-04 13:34:51,401 DEBUG lava-logs => PING(20) 2019-11-04 13:34:55,549 DEBUG Dhanu => PING(20) 2019-11-04 13:35:11,403 INFO scheduling health checks: 2019-11-04 13:35:11,432 INFO scheduling jobs: 2019-11-04 13:35:11,436 DEBUG - qemu 2019-11-04 13:35:11,468 DEBUG lava-logs => PING(20) 2019-11-04 13:35:15,565 DEBUG Dhanu => PING(20) 2019-11-04 13:35:31,468 INFO scheduling health checks: 2019-11-04 13:35:31,490 INFO scheduling jobs: 2019-11-04 13:35:31,492 DEBUG - qemu 2019-11-04 13:35:31,526 DEBUG lava-logs => PING(20) 2019-11-04 13:35:35,582 DEBUG Dhanu => PING(20) 2019-11-04 13:35:51,527 INFO scheduling health checks: 2019-11-04 13:35:51,548 INFO scheduling jobs: 2019-11-04 13:35:51,550 DEBUG - qemu 2019-11-04 13:35:51,580 DEBUG lava-logs => PING(20) 2019-11-04 13:35:55,603 DEBUG Dhanu => PING(20) 2019-11-04 13:36:11,581 INFO scheduling health checks: 2019-11-04 13:36:11,619 INFO scheduling jobs: 2019-11-04 13:36:11,623 DEBUG - qemu 2019-11-04 13:36:11,646 DEBUG lava-logs => PING(20) 2019-11-04 13:36:15,621 DEBUG Dhanu => PING(20) 2019-11-04 13:36:15,651 INFO scheduling health checks: 2019-11-04 13:36:15,674 INFO scheduling jobs: 2019-11-04 13:36:15,677 DEBUG - qemu 2019-11-04 13:36:15,770 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:36:15,770 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 799, in main_loop jobs = schedule(self.logger, self.events["available_dt"]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:36:15,819 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:36:17,370 INFO [INIT] Dropping privileges 2019-11-04 13:36:17,370 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:36:17,370 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:36:17,372 INFO [INIT] Marking all workers as offline 2019-11-04 13:36:17,440 INFO [INIT] Starting encryption 2019-11-04 13:36:17,442 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:36:17,443 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:36:17,444 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:36:17,457 INFO [INIT] LAVA master has started. 2019-11-04 13:36:17,457 INFO [INIT] Using protocol version 3 2019-11-04 13:36:31,661 DEBUG lava-logs => PING(20) 2019-11-04 13:36:35,639 DEBUG Dhanu => PING(20) 2019-11-04 13:36:35,639 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:36:37,458 INFO scheduling health checks: 2019-11-04 13:36:37,486 INFO scheduling jobs: 2019-11-04 13:36:37,487 DEBUG - qemu 2019-11-04 13:36:37,557 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:36:37,557 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:36:37,565 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:36:39,127 INFO [INIT] Dropping privileges 2019-11-04 13:36:39,127 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:36:39,127 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:36:39,129 INFO [INIT] Marking all workers as offline 2019-11-04 13:36:39,164 INFO [INIT] Starting encryption 2019-11-04 13:36:39,165 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:36:39,165 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:36:39,165 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:36:39,178 INFO [INIT] LAVA master has started. 2019-11-04 13:36:39,178 INFO [INIT] Using protocol version 3 2019-11-04 13:36:51,675 DEBUG lava-logs => PING(20) 2019-11-04 13:36:55,657 DEBUG Dhanu => PING(20) 2019-11-04 13:36:55,657 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:36:59,180 INFO scheduling health checks: 2019-11-04 13:36:59,210 INFO scheduling jobs: 2019-11-04 13:36:59,212 DEBUG - qemu 2019-11-04 13:36:59,288 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:36:59,288 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:36:59,293 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:37:00,610 INFO [INIT] Dropping privileges 2019-11-04 13:37:00,611 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:37:00,611 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:37:00,613 INFO [INIT] Marking all workers as offline 2019-11-04 13:37:00,655 INFO [INIT] Starting encryption 2019-11-04 13:37:00,658 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:37:00,659 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:37:00,659 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:37:00,672 INFO [INIT] LAVA master has started. 2019-11-04 13:37:00,672 INFO [INIT] Using protocol version 3 2019-11-04 13:37:11,688 DEBUG lava-logs => PING(20) 2019-11-04 13:37:15,677 DEBUG Dhanu => PING(20) 2019-11-04 13:37:15,677 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:37:20,676 INFO scheduling health checks: 2019-11-04 13:37:20,746 INFO scheduling jobs: 2019-11-04 13:37:20,750 DEBUG - qemu 2019-11-04 13:37:20,836 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:37:20,836 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:37:20,847 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:37:22,302 INFO [INIT] Dropping privileges 2019-11-04 13:37:22,302 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:37:22,303 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:37:22,304 INFO [INIT] Marking all workers as offline 2019-11-04 13:37:22,369 INFO [INIT] Starting encryption 2019-11-04 13:37:22,370 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:37:22,371 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:37:22,371 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:37:22,383 INFO [INIT] LAVA master has started. 2019-11-04 13:37:22,383 INFO [INIT] Using protocol version 3 2019-11-04 13:37:31,702 DEBUG lava-logs => PING(20) 2019-11-04 13:37:35,694 DEBUG Dhanu => PING(20) 2019-11-04 13:37:35,695 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:37:42,390 INFO scheduling health checks: 2019-11-04 13:37:42,426 INFO scheduling jobs: 2019-11-04 13:37:42,428 DEBUG - qemu 2019-11-04 13:37:42,503 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:37:42,503 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:37:42,507 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:37:43,680 INFO [INIT] Dropping privileges 2019-11-04 13:37:43,680 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:37:43,680 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:37:43,682 INFO [INIT] Marking all workers as offline 2019-11-04 13:37:43,715 INFO [INIT] Starting encryption 2019-11-04 13:37:43,718 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:37:43,718 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:37:43,719 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:37:43,736 INFO [INIT] LAVA master has started. 2019-11-04 13:37:43,736 INFO [INIT] Using protocol version 3 2019-11-04 13:37:51,723 DEBUG lava-logs => PING(20) 2019-11-04 13:37:55,715 DEBUG Dhanu => PING(20) 2019-11-04 13:37:55,716 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:38:03,741 INFO scheduling health checks: 2019-11-04 13:38:03,772 INFO scheduling jobs: 2019-11-04 13:38:03,774 DEBUG - qemu 2019-11-04 13:38:03,853 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:38:03,853 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:38:03,863 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:38:05,165 INFO [INIT] Dropping privileges 2019-11-04 13:38:05,165 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:38:05,166 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:38:05,167 INFO [INIT] Marking all workers as offline 2019-11-04 13:38:05,195 INFO [INIT] Starting encryption 2019-11-04 13:38:05,196 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:38:05,196 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:38:05,197 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:38:05,210 INFO [INIT] LAVA master has started. 2019-11-04 13:38:05,210 INFO [INIT] Using protocol version 3 2019-11-04 13:38:11,741 DEBUG lava-logs => PING(20) 2019-11-04 13:38:15,735 DEBUG Dhanu => PING(20) 2019-11-04 13:38:15,735 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:38:25,220 INFO scheduling health checks: 2019-11-04 13:38:25,263 INFO scheduling jobs: 2019-11-04 13:38:25,266 DEBUG - qemu 2019-11-04 13:38:25,339 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:38:25,339 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:38:25,353 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:38:26,667 INFO [INIT] Dropping privileges 2019-11-04 13:38:26,667 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:38:26,668 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:38:26,669 INFO [INIT] Marking all workers as offline 2019-11-04 13:38:26,697 INFO [INIT] Starting encryption 2019-11-04 13:38:26,698 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:38:26,699 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:38:26,699 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:38:26,714 INFO [INIT] LAVA master has started. 2019-11-04 13:38:26,714 INFO [INIT] Using protocol version 3 2019-11-04 13:38:31,758 DEBUG lava-logs => PING(20) 2019-11-04 13:38:35,754 DEBUG Dhanu => PING(20) 2019-11-04 13:38:35,754 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:38:46,726 INFO scheduling health checks: 2019-11-04 13:38:46,765 INFO scheduling jobs: 2019-11-04 13:38:46,768 DEBUG - qemu 2019-11-04 13:38:46,843 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:38:46,843 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:38:46,848 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:38:48,155 INFO [INIT] Dropping privileges 2019-11-04 13:38:48,155 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:38:48,156 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:38:48,157 INFO [INIT] Marking all workers as offline 2019-11-04 13:38:48,188 INFO [INIT] Starting encryption 2019-11-04 13:38:48,190 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:38:48,190 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:38:48,190 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:38:48,209 INFO [INIT] LAVA master has started. 2019-11-04 13:38:48,209 INFO [INIT] Using protocol version 3 2019-11-04 13:38:51,774 DEBUG lava-logs => PING(20) 2019-11-04 13:38:55,775 DEBUG Dhanu => PING(20) 2019-11-04 13:38:55,775 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:39:08,222 INFO scheduling health checks: 2019-11-04 13:39:08,256 INFO scheduling jobs: 2019-11-04 13:39:08,258 DEBUG - qemu 2019-11-04 13:39:08,327 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:39:08,328 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:39:08,336 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:39:09,439 INFO [INIT] Dropping privileges 2019-11-04 13:39:09,439 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:39:09,439 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:39:09,441 INFO [INIT] Marking all workers as offline 2019-11-04 13:39:09,468 INFO [INIT] Starting encryption 2019-11-04 13:39:09,470 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:39:09,470 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:39:09,471 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:39:09,488 INFO [INIT] LAVA master has started. 2019-11-04 13:39:09,488 INFO [INIT] Using protocol version 3 2019-11-04 13:39:11,789 DEBUG lava-logs => PING(20) 2019-11-04 13:39:15,792 DEBUG Dhanu => PING(20) 2019-11-04 13:39:15,793 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:39:29,490 INFO scheduling health checks: 2019-11-04 13:39:29,525 INFO scheduling jobs: 2019-11-04 13:39:29,527 DEBUG - qemu 2019-11-04 13:39:29,566 DEBUG Good ??? Bad (Invalid device configuration for qemu01) 2019-11-04 13:39:31,809 DEBUG lava-logs => PING(20) 2019-11-04 13:39:35,813 DEBUG Dhanu => PING(20) 2019-11-04 13:39:39,359 INFO [POLL] Received a signal, leaving 2019-11-04 13:39:39,359 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:39:40,344 INFO [INIT] Dropping privileges 2019-11-04 13:39:40,344 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:39:40,345 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:39:40,346 INFO [INIT] Marking all workers as offline 2019-11-04 13:39:40,382 INFO [INIT] Starting encryption 2019-11-04 13:39:40,384 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:39:40,385 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:39:40,385 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:39:40,401 INFO [INIT] LAVA master has started. 2019-11-04 13:39:40,401 INFO [INIT] Using protocol version 3 2019-11-04 13:39:43,708 INFO Dhanu => HELLO 2019-11-04 13:39:43,708 WARNING New dispatcher 2019-11-04 13:39:48,715 DEBUG Dhanu => PING(20) 2019-11-04 13:39:51,819 DEBUG lava-logs => PING(20) 2019-11-04 13:40:00,402 INFO scheduling health checks: 2019-11-04 13:40:00,407 INFO scheduling jobs: 2019-11-04 13:40:08,740 DEBUG Dhanu => PING(20) 2019-11-04 13:40:11,842 DEBUG lava-logs => PING(20) 2019-11-04 13:40:20,416 INFO scheduling health checks: 2019-11-04 13:40:20,426 INFO scheduling jobs: 2019-11-04 13:40:28,764 DEBUG Dhanu => PING(20) 2019-11-04 13:40:34,917 DEBUG lava-logs => PING(20) 2019-11-04 13:40:34,976 DEBUG lava-logs => PING(20) 2019-11-04 13:40:40,436 INFO scheduling health checks: 2019-11-04 13:40:40,447 INFO scheduling jobs: 2019-11-04 13:40:48,781 DEBUG Dhanu => PING(20) 2019-11-04 13:40:55,001 DEBUG lava-logs => PING(20) 2019-11-04 13:41:00,457 INFO scheduling health checks: 2019-11-04 13:41:00,469 INFO scheduling jobs: 2019-11-04 13:41:08,801 DEBUG Dhanu => PING(20) 2019-11-04 13:41:15,021 DEBUG lava-logs => PING(20) 2019-11-04 13:41:20,479 INFO scheduling health checks: 2019-11-04 13:41:20,489 INFO scheduling jobs: 2019-11-04 13:41:28,826 DEBUG Dhanu => PING(20) 2019-11-04 13:41:35,044 DEBUG lava-logs => PING(20) 2019-11-04 13:41:40,502 INFO scheduling health checks: 2019-11-04 13:41:40,512 INFO scheduling jobs: 2019-11-04 13:41:48,850 DEBUG Dhanu => PING(20) 2019-11-04 13:41:55,067 DEBUG lava-logs => PING(20) 2019-11-04 13:42:00,523 INFO scheduling health checks: 2019-11-04 13:42:00,534 INFO scheduling jobs: 2019-11-04 13:42:08,871 DEBUG Dhanu => PING(20) 2019-11-04 13:42:15,090 DEBUG lava-logs => PING(20) 2019-11-04 13:42:20,545 INFO scheduling health checks: 2019-11-04 13:42:20,557 INFO scheduling jobs: 2019-11-04 13:42:28,891 DEBUG Dhanu => PING(20) 2019-11-04 13:42:40,569 INFO scheduling health checks: 2019-11-04 13:42:40,580 INFO scheduling jobs: 2019-11-04 13:42:45,841 DEBUG lava-logs => PING(20) 2019-11-04 13:42:45,878 DEBUG lava-logs => PING(20) 2019-11-04 13:42:48,913 DEBUG Dhanu => PING(20) 2019-11-04 13:43:00,591 INFO scheduling health checks: 2019-11-04 13:43:00,602 INFO scheduling jobs: 2019-11-04 13:43:05,896 DEBUG lava-logs => PING(20) 2019-11-04 13:43:08,930 DEBUG Dhanu => PING(20) 2019-11-04 13:43:13,392 INFO Dhanu => HELLO 2019-11-04 13:43:13,393 WARNING Dispatcher has RESTARTED 2019-11-04 13:43:17,095 INFO [POLL] Received a signal, leaving 2019-11-04 13:43:17,095 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:43:18,075 INFO [INIT] Dropping privileges 2019-11-04 13:43:18,075 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:43:18,076 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:43:18,077 INFO [INIT] Marking all workers as offline 2019-11-04 13:43:18,115 INFO [INIT] Starting encryption 2019-11-04 13:43:18,116 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:43:18,117 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:43:18,117 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:43:18,133 INFO [INIT] LAVA master has started. 2019-11-04 13:43:18,133 INFO [INIT] Using protocol version 3 2019-11-04 13:43:18,395 DEBUG Dhanu => PING(20) 2019-11-04 13:43:18,396 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:43:25,914 DEBUG lava-logs => PING(20) 2019-11-04 13:43:38,144 INFO scheduling health checks: 2019-11-04 13:43:38,155 INFO scheduling jobs: 2019-11-04 13:43:38,413 DEBUG Dhanu => PING(20) 2019-11-04 13:43:45,932 DEBUG lava-logs => PING(20) 2019-11-04 13:43:58,173 INFO scheduling health checks: 2019-11-04 13:43:58,187 INFO scheduling jobs: 2019-11-04 13:43:58,431 DEBUG Dhanu => PING(20) 2019-11-04 13:44:05,950 DEBUG lava-logs => PING(20) 2019-11-04 13:44:18,201 INFO scheduling health checks: 2019-11-04 13:44:18,212 INFO scheduling jobs: 2019-11-04 13:44:18,450 DEBUG Dhanu => PING(20) 2019-11-04 13:44:25,969 DEBUG lava-logs => PING(20) 2019-11-04 13:44:38,230 INFO scheduling health checks: 2019-11-04 13:44:38,239 INFO scheduling jobs: 2019-11-04 13:44:38,471 DEBUG Dhanu => PING(20) 2019-11-04 13:44:45,992 DEBUG lava-logs => PING(20) 2019-11-04 13:44:58,251 INFO scheduling health checks: 2019-11-04 13:44:58,255 INFO scheduling jobs: 2019-11-04 13:44:58,491 DEBUG Dhanu => PING(20) 2019-11-04 13:44:59,549 INFO [POLL] Received a signal, leaving 2019-11-04 13:44:59,549 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:45:00,527 INFO [INIT] Dropping privileges 2019-11-04 13:45:00,527 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:45:00,527 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:45:00,529 INFO [INIT] Marking all workers as offline 2019-11-04 13:45:00,569 INFO [INIT] Starting encryption 2019-11-04 13:45:00,571 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:45:00,571 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:45:00,572 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:45:00,583 INFO [INIT] LAVA master has started. 2019-11-04 13:45:00,584 INFO [INIT] Using protocol version 3 2019-11-04 13:45:02,554 INFO Dhanu => HELLO 2019-11-04 13:45:02,555 WARNING New dispatcher 2019-11-04 13:45:06,009 DEBUG lava-logs => PING(20) 2019-11-04 13:45:07,558 DEBUG Dhanu => PING(20) 2019-11-04 13:45:20,594 INFO scheduling health checks: 2019-11-04 13:45:20,601 INFO scheduling jobs: 2019-11-04 13:45:26,031 DEBUG lava-logs => PING(20) 2019-11-04 13:45:27,580 DEBUG Dhanu => PING(20) 2019-11-04 13:45:40,616 INFO scheduling health checks: 2019-11-04 13:45:40,621 INFO scheduling jobs: 2019-11-04 13:45:46,052 DEBUG lava-logs => PING(20) 2019-11-04 13:45:47,601 DEBUG Dhanu => PING(20) 2019-11-04 13:46:00,638 INFO scheduling health checks: 2019-11-04 13:46:00,642 INFO scheduling jobs: 2019-11-04 13:46:06,075 DEBUG lava-logs => PING(20) 2019-11-04 13:46:07,626 DEBUG Dhanu => PING(20) 2019-11-04 13:46:20,657 INFO scheduling health checks: 2019-11-04 13:46:20,661 INFO scheduling jobs: 2019-11-04 13:46:26,096 DEBUG lava-logs => PING(20) 2019-11-04 13:46:27,649 DEBUG Dhanu => PING(20) 2019-11-04 13:46:40,674 INFO scheduling health checks: 2019-11-04 13:46:40,678 INFO scheduling jobs: 2019-11-04 13:46:46,118 DEBUG lava-logs => PING(20) 2019-11-04 13:46:47,669 DEBUG Dhanu => PING(20) 2019-11-04 13:47:00,694 INFO scheduling health checks: 2019-11-04 13:47:00,706 INFO scheduling jobs: 2019-11-04 13:47:06,137 DEBUG lava-logs => PING(20) 2019-11-04 13:47:07,686 DEBUG Dhanu => PING(20) 2019-11-04 13:47:20,720 INFO scheduling health checks: 2019-11-04 13:47:20,731 INFO scheduling jobs: 2019-11-04 13:47:26,153 DEBUG lava-logs => PING(20) 2019-11-04 13:47:27,711 DEBUG Dhanu => PING(20) 2019-11-04 13:47:40,743 INFO scheduling health checks: 2019-11-04 13:47:40,774 INFO scheduling jobs: 2019-11-04 13:47:40,776 DEBUG - qemu 2019-11-04 13:47:46,175 DEBUG lava-logs => PING(20) 2019-11-04 13:47:47,735 DEBUG Dhanu => PING(20) 2019-11-04 13:48:00,811 INFO scheduling health checks: 2019-11-04 13:48:00,831 INFO scheduling jobs: 2019-11-04 13:48:00,834 DEBUG - qemu 2019-11-04 13:48:06,196 DEBUG lava-logs => PING(20) 2019-11-04 13:48:07,759 DEBUG Dhanu => PING(20) 2019-11-04 13:48:20,862 INFO scheduling health checks: 2019-11-04 13:48:20,888 INFO scheduling jobs: 2019-11-04 13:48:20,891 DEBUG - qemu 2019-11-04 13:48:26,216 DEBUG lava-logs => PING(20) 2019-11-04 13:48:27,784 DEBUG Dhanu => PING(20) 2019-11-04 13:48:40,920 INFO scheduling health checks: 2019-11-04 13:48:40,946 INFO scheduling jobs: 2019-11-04 13:48:40,948 DEBUG - qemu 2019-11-04 13:48:46,228 DEBUG lava-logs => PING(20) 2019-11-04 13:48:47,802 DEBUG Dhanu => PING(20) 2019-11-04 13:49:00,965 INFO scheduling health checks: 2019-11-04 13:49:00,992 INFO scheduling jobs: 2019-11-04 13:49:00,994 DEBUG - qemu 2019-11-04 13:49:06,250 DEBUG lava-logs => PING(20) 2019-11-04 13:49:07,823 DEBUG Dhanu => PING(20) 2019-11-04 13:49:21,010 INFO scheduling health checks: 2019-11-04 13:49:21,027 INFO scheduling jobs: 2019-11-04 13:49:21,029 DEBUG - qemu 2019-11-04 13:49:26,270 DEBUG lava-logs => PING(20) 2019-11-04 13:49:27,842 DEBUG Dhanu => PING(20) 2019-11-04 13:49:41,052 INFO scheduling health checks: 2019-11-04 13:49:41,074 INFO scheduling jobs: 2019-11-04 13:49:41,075 DEBUG - qemu 2019-11-04 13:49:46,289 DEBUG lava-logs => PING(20) 2019-11-04 13:49:47,857 DEBUG Dhanu => PING(20) 2019-11-04 13:50:01,095 INFO scheduling health checks: 2019-11-04 13:50:01,133 INFO scheduling jobs: 2019-11-04 13:50:01,136 DEBUG - qemu 2019-11-04 13:50:06,313 DEBUG lava-logs => PING(20) 2019-11-04 13:50:07,874 DEBUG Dhanu => PING(20) 2019-11-04 13:50:21,168 INFO scheduling health checks: 2019-11-04 13:50:21,211 INFO scheduling jobs: 2019-11-04 13:50:21,215 DEBUG - qemu 2019-11-04 13:50:26,329 DEBUG lava-logs => PING(20) 2019-11-04 13:50:27,895 DEBUG Dhanu => PING(20) 2019-11-04 13:50:41,244 INFO scheduling health checks: 2019-11-04 13:50:41,269 INFO scheduling jobs: 2019-11-04 13:50:41,272 DEBUG - qemu 2019-11-04 13:50:46,352 DEBUG lava-logs => PING(20) 2019-11-04 13:50:47,920 DEBUG Dhanu => PING(20) 2019-11-04 13:51:01,303 INFO scheduling health checks: 2019-11-04 13:51:01,337 INFO scheduling jobs: 2019-11-04 13:51:01,339 DEBUG - qemu 2019-11-04 13:51:06,394 DEBUG lava-logs => PING(20) 2019-11-04 13:51:07,939 DEBUG Dhanu => PING(20) 2019-11-04 13:51:21,357 INFO scheduling health checks: 2019-11-04 13:51:21,384 INFO scheduling jobs: 2019-11-04 13:51:21,386 DEBUG - qemu 2019-11-04 13:51:26,413 DEBUG lava-logs => PING(20) 2019-11-04 13:51:27,959 DEBUG Dhanu => PING(20) 2019-11-04 13:51:41,402 INFO scheduling health checks: 2019-11-04 13:51:41,430 INFO scheduling jobs: 2019-11-04 13:51:41,432 DEBUG - qemu 2019-11-04 13:51:46,436 DEBUG lava-logs => PING(20) 2019-11-04 13:51:47,982 DEBUG Dhanu => PING(20) 2019-11-04 13:52:01,450 INFO scheduling health checks: 2019-11-04 13:52:01,477 INFO scheduling jobs: 2019-11-04 13:52:01,479 DEBUG - qemu 2019-11-04 13:52:06,453 DEBUG lava-logs => PING(20) 2019-11-04 13:52:08,002 DEBUG Dhanu => PING(20) 2019-11-04 13:52:21,511 INFO scheduling health checks: 2019-11-04 13:52:21,534 INFO scheduling jobs: 2019-11-04 13:52:21,537 DEBUG - qemu 2019-11-04 13:52:26,473 DEBUG lava-logs => PING(20) 2019-11-04 13:52:28,017 DEBUG Dhanu => PING(20) 2019-11-04 13:52:34,662 INFO scheduling health checks: 2019-11-04 13:52:34,691 INFO scheduling jobs: 2019-11-04 13:52:34,694 DEBUG - qemu 2019-11-04 13:52:34,769 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:52:34,769 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 799, in main_loop jobs = schedule(self.logger, self.events["available_dt"]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:52:34,775 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:52:35,953 INFO [INIT] Dropping privileges 2019-11-04 13:52:35,953 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:52:35,953 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:52:35,955 INFO [INIT] Marking all workers as offline 2019-11-04 13:52:36,003 INFO [INIT] Starting encryption 2019-11-04 13:52:36,005 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:52:36,005 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:52:36,006 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:52:36,023 INFO [INIT] LAVA master has started. 2019-11-04 13:52:36,024 INFO [INIT] Using protocol version 3 2019-11-04 13:52:46,485 DEBUG lava-logs => PING(20) 2019-11-04 13:52:48,029 DEBUG Dhanu => PING(20) 2019-11-04 13:52:48,030 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:52:56,029 INFO scheduling health checks: 2019-11-04 13:52:56,071 INFO scheduling jobs: 2019-11-04 13:52:56,076 DEBUG - qemu 2019-11-04 13:52:56,149 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:52:56,149 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:52:56,153 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:52:57,695 INFO [INIT] Dropping privileges 2019-11-04 13:52:57,695 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:52:57,696 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:52:57,698 INFO [INIT] Marking all workers as offline 2019-11-04 13:52:57,728 INFO [INIT] Starting encryption 2019-11-04 13:52:57,729 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:52:57,729 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:52:57,730 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:52:57,750 INFO [INIT] LAVA master has started. 2019-11-04 13:52:57,750 INFO [INIT] Using protocol version 3 2019-11-04 13:53:06,503 DEBUG lava-logs => PING(20) 2019-11-04 13:53:08,042 DEBUG Dhanu => PING(20) 2019-11-04 13:53:08,042 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:53:17,761 INFO scheduling health checks: 2019-11-04 13:53:17,796 INFO scheduling jobs: 2019-11-04 13:53:17,799 DEBUG - qemu 2019-11-04 13:53:17,882 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:53:17,883 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:53:17,887 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:53:19,169 INFO [INIT] Dropping privileges 2019-11-04 13:53:19,169 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:53:19,170 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:53:19,171 INFO [INIT] Marking all workers as offline 2019-11-04 13:53:19,207 INFO [INIT] Starting encryption 2019-11-04 13:53:19,209 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:53:19,210 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:53:19,210 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:53:19,226 INFO [INIT] LAVA master has started. 2019-11-04 13:53:19,227 INFO [INIT] Using protocol version 3 2019-11-04 13:53:26,517 DEBUG lava-logs => PING(20) 2019-11-04 13:53:28,061 DEBUG Dhanu => PING(20) 2019-11-04 13:53:28,062 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:53:39,239 INFO scheduling health checks: 2019-11-04 13:53:39,276 INFO scheduling jobs: 2019-11-04 13:53:39,279 DEBUG - qemu 2019-11-04 13:53:39,376 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:53:39,376 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:53:39,389 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:53:40,717 INFO [INIT] Dropping privileges 2019-11-04 13:53:40,717 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:53:40,718 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:53:40,719 INFO [INIT] Marking all workers as offline 2019-11-04 13:53:40,754 INFO [INIT] Starting encryption 2019-11-04 13:53:40,756 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:53:40,757 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:53:40,757 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:53:40,775 INFO [INIT] LAVA master has started. 2019-11-04 13:53:40,775 INFO [INIT] Using protocol version 3 2019-11-04 13:53:46,528 DEBUG lava-logs => PING(20) 2019-11-04 13:53:48,077 DEBUG Dhanu => PING(20) 2019-11-04 13:53:48,078 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:54:00,787 INFO scheduling health checks: 2019-11-04 13:54:00,822 INFO scheduling jobs: 2019-11-04 13:54:00,824 DEBUG - qemu 2019-11-04 13:54:00,899 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:54:00,900 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:54:00,904 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:54:02,158 INFO [INIT] Dropping privileges 2019-11-04 13:54:02,158 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:54:02,159 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:54:02,160 INFO [INIT] Marking all workers as offline 2019-11-04 13:54:02,257 INFO [INIT] Starting encryption 2019-11-04 13:54:02,260 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:54:02,260 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:54:02,261 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:54:02,276 INFO [INIT] LAVA master has started. 2019-11-04 13:54:02,276 INFO [INIT] Using protocol version 3 2019-11-04 13:54:06,541 DEBUG lava-logs => PING(20) 2019-11-04 13:54:08,094 DEBUG Dhanu => PING(20) 2019-11-04 13:54:08,095 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:54:22,291 INFO scheduling health checks: 2019-11-04 13:54:22,324 INFO scheduling jobs: 2019-11-04 13:54:22,327 DEBUG - qemu 2019-11-04 13:54:22,406 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:54:22,406 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:54:22,414 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:54:23,677 INFO [INIT] Dropping privileges 2019-11-04 13:54:23,677 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:54:23,678 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:54:23,679 INFO [INIT] Marking all workers as offline 2019-11-04 13:54:23,714 INFO [INIT] Starting encryption 2019-11-04 13:54:23,715 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:54:23,716 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:54:23,716 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:54:23,732 INFO [INIT] LAVA master has started. 2019-11-04 13:54:23,732 INFO [INIT] Using protocol version 3 2019-11-04 13:54:26,556 DEBUG lava-logs => PING(20) 2019-11-04 13:54:28,114 DEBUG Dhanu => PING(20) 2019-11-04 13:54:28,114 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:54:43,744 INFO scheduling health checks: 2019-11-04 13:54:43,782 INFO scheduling jobs: 2019-11-04 13:54:43,784 DEBUG - qemu 2019-11-04 13:54:43,858 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:54:43,858 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:54:43,862 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:54:45,161 INFO [INIT] Dropping privileges 2019-11-04 13:54:45,161 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:54:45,161 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:54:45,163 INFO [INIT] Marking all workers as offline 2019-11-04 13:54:45,194 INFO [INIT] Starting encryption 2019-11-04 13:54:45,196 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:54:45,197 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:54:45,197 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:54:45,211 INFO [INIT] LAVA master has started. 2019-11-04 13:54:45,211 INFO [INIT] Using protocol version 3 2019-11-04 13:54:46,567 DEBUG lava-logs => PING(20) 2019-11-04 13:54:48,132 DEBUG Dhanu => PING(20) 2019-11-04 13:54:48,133 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:55:05,228 INFO scheduling health checks: 2019-11-04 13:55:05,262 INFO scheduling jobs: 2019-11-04 13:55:05,263 DEBUG - qemu 2019-11-04 13:55:05,336 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:55:05,336 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:55:05,341 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:55:06,678 INFO [INIT] Dropping privileges 2019-11-04 13:55:06,678 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:55:06,678 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:55:06,680 INFO [INIT] Marking all workers as offline 2019-11-04 13:55:06,707 INFO [INIT] Starting encryption 2019-11-04 13:55:06,709 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:55:06,709 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:55:06,710 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:55:06,722 INFO [INIT] LAVA master has started. 2019-11-04 13:55:06,722 INFO [INIT] Using protocol version 3 2019-11-04 13:55:06,832 DEBUG lava-logs => PING(20) 2019-11-04 13:55:08,153 DEBUG Dhanu => PING(20) 2019-11-04 13:55:08,154 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:55:26,741 INFO scheduling health checks: 2019-11-04 13:55:26,775 INFO scheduling jobs: 2019-11-04 13:55:26,777 DEBUG - qemu 2019-11-04 13:55:26,846 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:55:26,846 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:55:26,854 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:55:28,158 INFO [INIT] Dropping privileges 2019-11-04 13:55:28,158 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:55:28,158 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:55:28,160 INFO [INIT] Marking all workers as offline 2019-11-04 13:55:28,198 INFO [INIT] Starting encryption 2019-11-04 13:55:28,200 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:55:28,200 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:55:28,201 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:55:28,218 INFO [INIT] LAVA master has started. 2019-11-04 13:55:28,219 INFO [INIT] Using protocol version 3 2019-11-04 13:55:28,354 DEBUG Dhanu => PING(20) 2019-11-04 13:55:28,355 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:55:46,864 DEBUG lava-logs => PING(20) 2019-11-04 13:55:48,220 INFO scheduling health checks: 2019-11-04 13:55:48,255 INFO scheduling jobs: 2019-11-04 13:55:48,257 DEBUG - qemu 2019-11-04 13:55:48,330 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:55:48,330 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 401, in __getitem__ if self._queue[-1] != key: RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:55:48,337 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:55:49,532 INFO [INIT] Dropping privileges 2019-11-04 13:55:49,533 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:55:49,533 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:55:49,535 INFO [INIT] Marking all workers as offline 2019-11-04 13:55:49,567 INFO [INIT] Starting encryption 2019-11-04 13:55:49,568 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:55:49,568 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:55:49,569 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:55:49,582 INFO [INIT] LAVA master has started. 2019-11-04 13:55:49,582 INFO [INIT] Using protocol version 3 2019-11-04 13:55:49,652 DEBUG Dhanu => PING(20) 2019-11-04 13:55:49,653 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:55:52,780 INFO Dhanu => HELLO 2019-11-04 13:55:52,781 WARNING Dispatcher has RESTARTED 2019-11-04 13:55:57,786 DEBUG Dhanu => PING(20) 2019-11-04 13:56:06,884 DEBUG lava-logs => PING(20) 2019-11-04 13:56:09,584 INFO scheduling health checks: 2019-11-04 13:56:09,623 INFO scheduling jobs: 2019-11-04 13:56:09,626 DEBUG - qemu 2019-11-04 13:56:09,706 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:56:09,706 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 401, in __getitem__ if self._queue[-1] != key: RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:56:09,715 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:56:11,059 INFO [INIT] Dropping privileges 2019-11-04 13:56:11,059 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:56:11,059 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:56:11,062 INFO [INIT] Marking all workers as offline 2019-11-04 13:56:11,102 INFO [INIT] Starting encryption 2019-11-04 13:56:11,103 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:56:11,103 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:56:11,103 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:56:11,114 INFO [INIT] LAVA master has started. 2019-11-04 13:56:11,114 INFO [INIT] Using protocol version 3 2019-11-04 13:56:17,807 DEBUG Dhanu => PING(20) 2019-11-04 13:56:17,807 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:56:26,902 DEBUG lava-logs => PING(20) 2019-11-04 13:56:31,119 INFO scheduling health checks: 2019-11-04 13:56:31,148 INFO scheduling jobs: 2019-11-04 13:56:31,150 DEBUG - qemu 2019-11-04 13:56:31,220 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:56:31,221 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 401, in __getitem__ if self._queue[-1] != key: RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:56:31,227 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:56:32,552 INFO [INIT] Dropping privileges 2019-11-04 13:56:32,552 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:56:32,553 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:56:32,554 INFO [INIT] Marking all workers as offline 2019-11-04 13:56:32,594 INFO [INIT] Starting encryption 2019-11-04 13:56:32,596 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:56:32,596 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:56:32,596 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:56:32,610 INFO [INIT] LAVA master has started. 2019-11-04 13:56:32,611 INFO [INIT] Using protocol version 3 2019-11-04 13:56:37,824 DEBUG Dhanu => PING(20) 2019-11-04 13:56:37,825 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:56:46,920 DEBUG lava-logs => PING(20) 2019-11-04 13:56:52,617 INFO scheduling health checks: 2019-11-04 13:56:52,650 INFO scheduling jobs: 2019-11-04 13:56:52,652 DEBUG - qemu 2019-11-04 13:56:52,721 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:56:52,722 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 401, in __getitem__ if self._queue[-1] != key: RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:56:52,733 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:56:53,918 INFO [INIT] Dropping privileges 2019-11-04 13:56:53,918 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:56:53,918 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:56:53,920 INFO [INIT] Marking all workers as offline 2019-11-04 13:56:53,951 INFO [INIT] Starting encryption 2019-11-04 13:56:53,953 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:56:53,953 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:56:53,954 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:56:53,968 INFO [INIT] LAVA master has started. 2019-11-04 13:56:53,969 INFO [INIT] Using protocol version 3 2019-11-04 13:56:57,842 DEBUG Dhanu => PING(20) 2019-11-04 13:56:57,842 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:57:06,937 DEBUG lava-logs => PING(20) 2019-11-04 13:57:13,976 INFO scheduling health checks: 2019-11-04 13:57:14,019 INFO scheduling jobs: 2019-11-04 13:57:14,022 DEBUG - qemu 2019-11-04 13:57:14,095 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:57:14,095 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 401, in __getitem__ if self._queue[-1] != key: RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:57:14,107 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:57:15,411 INFO [INIT] Dropping privileges 2019-11-04 13:57:15,411 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:57:15,412 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:57:15,413 INFO [INIT] Marking all workers as offline 2019-11-04 13:57:15,453 INFO [INIT] Starting encryption 2019-11-04 13:57:15,456 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:57:15,456 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:57:15,457 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:57:15,470 INFO [INIT] LAVA master has started. 2019-11-04 13:57:15,470 INFO [INIT] Using protocol version 3 2019-11-04 13:57:17,863 DEBUG Dhanu => PING(20) 2019-11-04 13:57:17,863 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:57:26,952 DEBUG lava-logs => PING(20) 2019-11-04 13:57:35,479 INFO scheduling health checks: 2019-11-04 13:57:35,521 INFO scheduling jobs: 2019-11-04 13:57:35,524 DEBUG - qemu 2019-11-04 13:57:35,598 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:57:35,599 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 401, in __getitem__ if self._queue[-1] != key: RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:57:35,609 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:57:36,938 INFO [INIT] Dropping privileges 2019-11-04 13:57:36,938 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:57:36,938 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:57:36,940 INFO [INIT] Marking all workers as offline 2019-11-04 13:57:37,000 INFO [INIT] Starting encryption 2019-11-04 13:57:37,002 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:57:37,003 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:57:37,003 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:57:37,019 INFO [INIT] LAVA master has started. 2019-11-04 13:57:37,020 INFO [INIT] Using protocol version 3 2019-11-04 13:57:37,882 DEBUG Dhanu => PING(20) 2019-11-04 13:57:37,882 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:57:46,974 DEBUG lava-logs => PING(20) 2019-11-04 13:57:57,030 INFO scheduling health checks: 2019-11-04 13:57:57,056 INFO scheduling jobs: 2019-11-04 13:57:57,058 DEBUG - qemu 2019-11-04 13:57:57,133 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:57:57,133 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:57:57,138 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:57:58,404 INFO [INIT] Dropping privileges 2019-11-04 13:57:58,404 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:57:58,405 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:57:58,406 INFO [INIT] Marking all workers as offline 2019-11-04 13:57:58,435 INFO [INIT] Starting encryption 2019-11-04 13:57:58,437 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:57:58,438 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:57:58,438 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:57:58,452 INFO [INIT] LAVA master has started. 2019-11-04 13:57:58,452 INFO [INIT] Using protocol version 3 2019-11-04 13:57:58,598 DEBUG Dhanu => PING(20) 2019-11-04 13:57:58,599 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:58:05,668 INFO [POLL] Received a signal, leaving 2019-11-04 13:58:05,668 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:58:06,647 INFO [INIT] Dropping privileges 2019-11-04 13:58:06,647 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:58:06,647 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:58:06,649 INFO [INIT] Marking all workers as offline 2019-11-04 13:58:06,680 INFO [INIT] Starting encryption 2019-11-04 13:58:06,681 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:58:06,682 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:58:06,682 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:58:06,699 INFO [INIT] LAVA master has started. 2019-11-04 13:58:06,699 INFO [INIT] Using protocol version 3 2019-11-04 13:58:06,987 DEBUG lava-logs => PING(20) 2019-11-04 13:58:08,746 INFO Dhanu => HELLO 2019-11-04 13:58:08,746 WARNING New dispatcher 2019-11-04 13:58:13,753 DEBUG Dhanu => PING(20) 2019-11-04 13:58:26,707 INFO scheduling health checks: 2019-11-04 13:58:26,748 INFO scheduling jobs: 2019-11-04 13:58:26,751 DEBUG - qemu 2019-11-04 13:58:26,831 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:58:26,832 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:58:26,836 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:58:28,163 INFO [INIT] Dropping privileges 2019-11-04 13:58:28,163 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:58:28,164 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:58:28,165 INFO [INIT] Marking all workers as offline 2019-11-04 13:58:28,194 INFO [INIT] Starting encryption 2019-11-04 13:58:28,196 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:58:28,196 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:58:28,197 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:58:28,215 INFO [INIT] LAVA master has started. 2019-11-04 13:58:28,215 INFO [INIT] Using protocol version 3 2019-11-04 13:58:28,246 DEBUG lava-logs => PING(20) 2019-11-04 13:58:33,766 DEBUG Dhanu => PING(20) 2019-11-04 13:58:33,766 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:58:44,024 DEBUG lava-logs => PING(20) 2019-11-04 13:58:44,039 DEBUG lava-logs => PING(20) 2019-11-04 13:58:48,219 INFO scheduling health checks: 2019-11-04 13:58:48,261 INFO scheduling jobs: 2019-11-04 13:58:48,263 DEBUG - qemu 2019-11-04 13:58:48,338 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:58:48,339 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:58:48,344 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:58:49,736 INFO [INIT] Dropping privileges 2019-11-04 13:58:49,737 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:58:49,737 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:58:49,739 INFO [INIT] Marking all workers as offline 2019-11-04 13:58:49,774 INFO [INIT] Starting encryption 2019-11-04 13:58:49,776 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:58:49,776 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:58:49,777 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:58:49,792 INFO [INIT] LAVA master has started. 2019-11-04 13:58:49,792 INFO [INIT] Using protocol version 3 2019-11-04 13:58:53,781 DEBUG Dhanu => PING(20) 2019-11-04 13:58:53,781 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:59:04,054 DEBUG lava-logs => PING(20) 2019-11-04 13:59:09,798 INFO scheduling health checks: 2019-11-04 13:59:09,830 INFO scheduling jobs: 2019-11-04 13:59:09,834 DEBUG - qemu 2019-11-04 13:59:09,911 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:59:09,911 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:59:09,916 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:59:12,515 INFO [INIT] Dropping privileges 2019-11-04 13:59:12,515 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:59:12,516 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:59:12,517 INFO [INIT] Marking all workers as offline 2019-11-04 13:59:12,587 INFO [INIT] Starting encryption 2019-11-04 13:59:12,589 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:59:12,589 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:59:12,590 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:59:12,611 INFO [INIT] LAVA master has started. 2019-11-04 13:59:12,611 INFO [INIT] Using protocol version 3 2019-11-04 13:59:13,807 DEBUG Dhanu => PING(20) 2019-11-04 13:59:13,808 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:59:24,069 DEBUG lava-logs => PING(20) 2019-11-04 13:59:32,619 INFO scheduling health checks: 2019-11-04 13:59:32,666 INFO scheduling jobs: 2019-11-04 13:59:32,669 DEBUG - qemu 2019-11-04 13:59:32,766 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:59:32,766 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:59:32,773 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:59:34,371 INFO [INIT] Dropping privileges 2019-11-04 13:59:34,371 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:59:34,372 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:59:34,373 INFO [INIT] Marking all workers as offline 2019-11-04 13:59:34,402 INFO [INIT] Starting encryption 2019-11-04 13:59:34,403 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:59:34,403 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:59:34,403 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:59:34,419 INFO [INIT] LAVA master has started. 2019-11-04 13:59:34,419 INFO [INIT] Using protocol version 3 2019-11-04 13:59:34,499 DEBUG Dhanu => PING(20) 2019-11-04 13:59:34,500 WARNING Unknown dispatcher (server crashed) 2019-11-04 13:59:44,086 DEBUG lava-logs => PING(20) 2019-11-04 13:59:54,430 INFO scheduling health checks: 2019-11-04 13:59:54,457 INFO scheduling jobs: 2019-11-04 13:59:54,459 DEBUG - qemu 2019-11-04 13:59:54,534 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 13:59:54,534 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 13:59:54,539 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 13:59:55,671 INFO [INIT] Dropping privileges 2019-11-04 13:59:55,672 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 13:59:55,672 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 13:59:55,674 INFO [INIT] Marking all workers as offline 2019-11-04 13:59:55,702 INFO [INIT] Starting encryption 2019-11-04 13:59:55,704 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 13:59:55,705 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:59:55,705 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 13:59:55,720 INFO [INIT] LAVA master has started. 2019-11-04 13:59:55,720 INFO [INIT] Using protocol version 3 2019-11-04 14:00:04,107 DEBUG lava-logs => PING(20) 2019-11-04 14:00:14,538 DEBUG Dhanu => PING(20) 2019-11-04 14:00:14,539 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:00:15,721 INFO scheduling health checks: 2019-11-04 14:00:15,748 INFO scheduling jobs: 2019-11-04 14:00:15,749 DEBUG - qemu 2019-11-04 14:00:15,847 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:00:15,847 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:00:15,854 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:00:17,298 INFO [INIT] Dropping privileges 2019-11-04 14:00:17,298 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:00:17,298 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:00:17,301 INFO [INIT] Marking all workers as offline 2019-11-04 14:00:17,326 INFO [INIT] Starting encryption 2019-11-04 14:00:17,327 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:00:17,327 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:00:17,327 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:00:17,339 INFO [INIT] LAVA master has started. 2019-11-04 14:00:17,339 INFO [INIT] Using protocol version 3 2019-11-04 14:00:24,122 DEBUG lava-logs => PING(20) 2019-11-04 14:00:34,553 DEBUG Dhanu => PING(20) 2019-11-04 14:00:34,553 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:00:37,340 INFO scheduling health checks: 2019-11-04 14:00:37,385 INFO scheduling jobs: 2019-11-04 14:00:37,388 DEBUG - qemu 2019-11-04 14:00:37,488 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:00:37,489 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:00:37,494 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:00:39,021 INFO [INIT] Dropping privileges 2019-11-04 14:00:39,021 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:00:39,022 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:00:39,025 INFO [INIT] Marking all workers as offline 2019-11-04 14:00:39,061 INFO [INIT] Starting encryption 2019-11-04 14:00:39,062 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:00:39,062 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:00:39,063 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:00:39,075 INFO [INIT] LAVA master has started. 2019-11-04 14:00:39,075 INFO [INIT] Using protocol version 3 2019-11-04 14:00:44,137 DEBUG lava-logs => PING(20) 2019-11-04 14:00:54,571 DEBUG Dhanu => PING(20) 2019-11-04 14:00:54,572 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:00:59,080 INFO scheduling health checks: 2019-11-04 14:00:59,119 INFO scheduling jobs: 2019-11-04 14:00:59,121 DEBUG - qemu 2019-11-04 14:00:59,204 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:00:59,204 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:00:59,215 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:01:00,540 INFO [INIT] Dropping privileges 2019-11-04 14:01:00,540 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:01:00,540 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:01:00,542 INFO [INIT] Marking all workers as offline 2019-11-04 14:01:00,577 INFO [INIT] Starting encryption 2019-11-04 14:01:00,579 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:01:00,579 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:01:00,580 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:01:00,595 INFO [INIT] LAVA master has started. 2019-11-04 14:01:00,595 INFO [INIT] Using protocol version 3 2019-11-04 14:01:04,154 DEBUG lava-logs => PING(20) 2019-11-04 14:01:14,589 DEBUG Dhanu => PING(20) 2019-11-04 14:01:14,590 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:01:20,601 INFO scheduling health checks: 2019-11-04 14:01:20,632 INFO scheduling jobs: 2019-11-04 14:01:20,634 DEBUG - qemu 2019-11-04 14:01:20,745 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:01:20,745 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:01:20,756 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:01:22,075 INFO [INIT] Dropping privileges 2019-11-04 14:01:22,075 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:01:22,076 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:01:22,078 INFO [INIT] Marking all workers as offline 2019-11-04 14:01:22,110 INFO [INIT] Starting encryption 2019-11-04 14:01:22,111 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:01:22,111 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:01:22,111 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:01:22,120 INFO [INIT] LAVA master has started. 2019-11-04 14:01:22,120 INFO [INIT] Using protocol version 3 2019-11-04 14:01:24,165 DEBUG lava-logs => PING(20) 2019-11-04 14:01:34,602 DEBUG Dhanu => PING(20) 2019-11-04 14:01:34,602 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:01:42,128 INFO scheduling health checks: 2019-11-04 14:01:42,168 INFO scheduling jobs: 2019-11-04 14:01:42,171 DEBUG - qemu 2019-11-04 14:01:42,245 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:01:42,245 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:01:42,250 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:01:43,416 INFO [INIT] Dropping privileges 2019-11-04 14:01:43,416 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:01:43,416 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:01:43,418 INFO [INIT] Marking all workers as offline 2019-11-04 14:01:43,446 INFO [INIT] Starting encryption 2019-11-04 14:01:43,448 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:01:43,448 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:01:43,449 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:01:43,463 INFO [INIT] LAVA master has started. 2019-11-04 14:01:43,463 INFO [INIT] Using protocol version 3 2019-11-04 14:01:44,175 DEBUG lava-logs => PING(20) 2019-11-04 14:01:54,614 DEBUG Dhanu => PING(20) 2019-11-04 14:01:54,614 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:02:03,472 INFO scheduling health checks: 2019-11-04 14:02:03,503 INFO scheduling jobs: 2019-11-04 14:02:03,507 DEBUG - qemu 2019-11-04 14:02:03,583 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:02:03,583 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 3, in top-level template code {% set memory = '1024' %} [Previous line repeated 970 more times] File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:02:03,588 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:02:04,961 INFO [INIT] Dropping privileges 2019-11-04 14:02:04,961 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:02:04,961 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:02:04,963 INFO [INIT] Marking all workers as offline 2019-11-04 14:02:04,994 INFO [INIT] Starting encryption 2019-11-04 14:02:04,995 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:02:04,996 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:02:04,996 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:02:05,011 INFO [INIT] LAVA master has started. 2019-11-04 14:02:05,011 INFO [INIT] Using protocol version 3 2019-11-04 14:02:05,080 DEBUG lava-logs => PING(20) 2019-11-04 14:02:14,632 DEBUG Dhanu => PING(20) 2019-11-04 14:02:14,632 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:02:25,022 INFO scheduling health checks: 2019-11-04 14:02:25,072 INFO scheduling jobs: 2019-11-04 14:02:25,075 DEBUG - qemu 2019-11-04 14:02:25,111 DEBUG Good ??? Bad (Invalid device configuration for qemu01) 2019-11-04 14:02:25,124 DEBUG lava-logs => PING(20) 2019-11-04 14:02:34,501 INFO Dhanu => HELLO 2019-11-04 14:02:34,501 WARNING Dispatcher has RESTARTED 2019-11-04 14:02:39,508 DEBUG Dhanu => PING(20) 2019-11-04 14:02:45,125 INFO scheduling health checks: 2019-11-04 14:02:45,137 INFO scheduling jobs: 2019-11-04 14:02:46,736 DEBUG lava-logs => PING(20) 2019-11-04 14:02:46,752 DEBUG lava-logs => PING(20) 2019-11-04 14:02:59,527 DEBUG Dhanu => PING(20) 2019-11-04 14:03:05,149 INFO scheduling health checks: 2019-11-04 14:03:05,159 INFO scheduling jobs: 2019-11-04 14:03:06,775 DEBUG lava-logs => PING(20) 2019-11-04 14:03:06,900 INFO [POLL] Received a signal, leaving 2019-11-04 14:03:06,900 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:03:07,867 INFO [INIT] Dropping privileges 2019-11-04 14:03:07,867 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:03:07,868 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:03:07,869 INFO [INIT] Marking all workers as offline 2019-11-04 14:03:07,899 INFO [INIT] Starting encryption 2019-11-04 14:03:07,901 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:03:07,901 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:03:07,902 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:03:07,915 INFO [INIT] LAVA master has started. 2019-11-04 14:03:07,915 INFO [INIT] Using protocol version 3 2019-11-04 14:03:19,547 DEBUG Dhanu => PING(20) 2019-11-04 14:03:19,548 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:03:26,796 DEBUG lava-logs => PING(20) 2019-11-04 14:03:27,916 INFO scheduling health checks: 2019-11-04 14:03:27,932 INFO scheduling jobs: 2019-11-04 14:03:39,566 DEBUG Dhanu => PING(20) 2019-11-04 14:03:46,819 DEBUG lava-logs => PING(20) 2019-11-04 14:03:47,941 INFO scheduling health checks: 2019-11-04 14:03:47,951 INFO scheduling jobs: 2019-11-04 14:03:59,590 DEBUG Dhanu => PING(20) 2019-11-04 14:04:06,842 DEBUG lava-logs => PING(20) 2019-11-04 14:04:07,962 INFO scheduling health checks: 2019-11-04 14:04:07,973 INFO scheduling jobs: 2019-11-04 14:04:19,609 DEBUG Dhanu => PING(20) 2019-11-04 14:04:26,861 DEBUG lava-logs => PING(20) 2019-11-04 14:04:27,985 INFO scheduling health checks: 2019-11-04 14:04:27,995 INFO scheduling jobs: 2019-11-04 14:04:39,629 DEBUG Dhanu => PING(20) 2019-11-04 14:04:46,880 DEBUG lava-logs => PING(20) 2019-11-04 14:04:48,004 INFO scheduling health checks: 2019-11-04 14:04:48,014 INFO scheduling jobs: 2019-11-04 14:04:59,652 DEBUG Dhanu => PING(20) 2019-11-04 14:05:06,896 DEBUG lava-logs => PING(20) 2019-11-04 14:05:08,025 INFO scheduling health checks: 2019-11-04 14:05:08,035 INFO scheduling jobs: 2019-11-04 14:05:19,677 DEBUG Dhanu => PING(20) 2019-11-04 14:05:26,912 DEBUG lava-logs => PING(20) 2019-11-04 14:05:28,047 INFO scheduling health checks: 2019-11-04 14:05:28,058 INFO scheduling jobs: 2019-11-04 14:05:39,701 DEBUG Dhanu => PING(20) 2019-11-04 14:05:46,934 DEBUG lava-logs => PING(20) 2019-11-04 14:05:48,074 INFO scheduling health checks: 2019-11-04 14:05:48,086 INFO scheduling jobs: 2019-11-04 14:05:56,311 INFO [POLL] Received a signal, leaving 2019-11-04 14:05:56,312 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:05:57,313 INFO [INIT] Dropping privileges 2019-11-04 14:05:57,313 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:05:57,313 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:05:57,315 INFO [INIT] Marking all workers as offline 2019-11-04 14:05:57,349 INFO [INIT] Starting encryption 2019-11-04 14:05:57,351 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:05:57,351 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:05:57,352 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:05:57,367 INFO [INIT] LAVA master has started. 2019-11-04 14:05:57,367 INFO [INIT] Using protocol version 3 2019-11-04 14:05:59,721 DEBUG Dhanu => PING(20) 2019-11-04 14:05:59,721 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:06:06,956 DEBUG lava-logs => PING(20) 2019-11-04 14:06:17,374 INFO scheduling health checks: 2019-11-04 14:06:17,388 INFO scheduling jobs: 2019-11-04 14:06:19,743 DEBUG Dhanu => PING(20) 2019-11-04 14:06:21,658 DEBUG lava-logs => PING(20) 2019-11-04 14:06:21,684 DEBUG lava-logs => PING(20) 2019-11-04 14:06:34,356 INFO Dhanu => HELLO 2019-11-04 14:06:34,356 WARNING Dispatcher has RESTARTED 2019-11-04 14:06:37,399 INFO scheduling health checks: 2019-11-04 14:06:37,408 INFO scheduling jobs: 2019-11-04 14:06:39,361 DEBUG Dhanu => PING(20) 2019-11-04 14:06:41,706 DEBUG lava-logs => PING(20) 2019-11-04 14:06:57,422 INFO scheduling health checks: 2019-11-04 14:06:57,427 INFO scheduling jobs: 2019-11-04 14:06:59,377 DEBUG Dhanu => PING(20) 2019-11-04 14:07:01,986 DEBUG lava-logs => PING(20) 2019-11-04 14:07:17,438 INFO scheduling health checks: 2019-11-04 14:07:17,449 INFO scheduling jobs: 2019-11-04 14:07:19,489 DEBUG Dhanu => PING(20) 2019-11-04 14:07:22,006 DEBUG lava-logs => PING(20) 2019-11-04 14:07:37,459 INFO scheduling health checks: 2019-11-04 14:07:37,464 INFO scheduling jobs: 2019-11-04 14:07:39,525 DEBUG Dhanu => PING(20) 2019-11-04 14:07:42,029 DEBUG lava-logs => PING(20) 2019-11-04 14:07:57,472 INFO scheduling health checks: 2019-11-04 14:07:57,482 INFO scheduling jobs: 2019-11-04 14:07:59,541 DEBUG Dhanu => PING(20) 2019-11-04 14:08:02,052 DEBUG lava-logs => PING(20) 2019-11-04 14:08:17,499 INFO scheduling health checks: 2019-11-04 14:08:17,508 INFO scheduling jobs: 2019-11-04 14:08:19,565 DEBUG Dhanu => PING(20) 2019-11-04 14:08:22,074 DEBUG lava-logs => PING(20) 2019-11-04 14:08:37,523 INFO scheduling health checks: 2019-11-04 14:08:37,534 INFO scheduling jobs: 2019-11-04 14:08:39,585 DEBUG Dhanu => PING(20) 2019-11-04 14:08:42,094 DEBUG lava-logs => PING(20) 2019-11-04 14:08:57,544 INFO scheduling health checks: 2019-11-04 14:08:57,555 INFO scheduling jobs: 2019-11-04 14:08:59,606 DEBUG Dhanu => PING(20) 2019-11-04 14:09:02,114 DEBUG lava-logs => PING(20) 2019-11-04 14:09:17,566 INFO scheduling health checks: 2019-11-04 14:09:17,576 INFO scheduling jobs: 2019-11-04 14:09:19,627 DEBUG Dhanu => PING(20) 2019-11-04 14:09:22,132 DEBUG lava-logs => PING(20) 2019-11-04 14:09:29,913 INFO Dhanu => HELLO 2019-11-04 14:09:29,914 WARNING Dispatcher has RESTARTED 2019-11-04 14:09:34,918 DEBUG Dhanu => PING(20) 2019-11-04 14:09:37,590 INFO scheduling health checks: 2019-11-04 14:09:37,601 INFO scheduling jobs: 2019-11-04 14:09:47,594 DEBUG lava-logs => PING(20) 2019-11-04 14:09:47,626 DEBUG lava-logs => PING(20) 2019-11-04 14:09:54,939 DEBUG Dhanu => PING(20) 2019-11-04 14:09:55,018 INFO [POLL] Received a signal, leaving 2019-11-04 14:09:55,018 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:09:56,457 INFO [INIT] Dropping privileges 2019-11-04 14:09:56,457 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:09:56,458 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:09:56,459 INFO [INIT] Marking all workers as offline 2019-11-04 14:09:56,483 INFO [INIT] Starting encryption 2019-11-04 14:09:56,484 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:09:56,484 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:09:56,485 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:09:56,499 INFO [INIT] LAVA master has started. 2019-11-04 14:09:56,499 INFO [INIT] Using protocol version 3 2019-11-04 14:10:07,638 DEBUG lava-logs => PING(20) 2019-11-04 14:10:14,954 DEBUG Dhanu => PING(20) 2019-11-04 14:10:14,954 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:10:16,500 INFO scheduling health checks: 2019-11-04 14:10:16,506 INFO scheduling jobs: 2019-11-04 14:10:27,659 DEBUG lava-logs => PING(20) 2019-11-04 14:10:34,973 DEBUG Dhanu => PING(20) 2019-11-04 14:10:36,515 INFO scheduling health checks: 2019-11-04 14:10:36,519 INFO scheduling jobs: 2019-11-04 14:10:47,924 DEBUG lava-logs => PING(20) 2019-11-04 14:10:54,990 DEBUG Dhanu => PING(20) 2019-11-04 14:10:56,532 INFO scheduling health checks: 2019-11-04 14:10:56,538 INFO scheduling jobs: 2019-11-04 14:11:07,949 DEBUG lava-logs => PING(20) 2019-11-04 14:11:15,011 DEBUG Dhanu => PING(20) 2019-11-04 14:11:16,551 INFO scheduling health checks: 2019-11-04 14:11:16,562 INFO scheduling jobs: 2019-11-04 14:11:27,970 DEBUG lava-logs => PING(20) 2019-11-04 14:11:35,033 DEBUG Dhanu => PING(20) 2019-11-04 14:11:36,573 INFO scheduling health checks: 2019-11-04 14:11:36,615 INFO scheduling jobs: 2019-11-04 14:11:36,617 DEBUG - qemu 2019-11-04 14:11:47,989 DEBUG lava-logs => PING(20) 2019-11-04 14:11:55,051 DEBUG Dhanu => PING(20) 2019-11-04 14:11:56,647 INFO scheduling health checks: 2019-11-04 14:11:56,661 INFO scheduling jobs: 2019-11-04 14:11:56,663 DEBUG - qemu 2019-11-04 14:12:08,012 DEBUG lava-logs => PING(20) 2019-11-04 14:12:15,089 DEBUG Dhanu => PING(20) 2019-11-04 14:12:16,679 INFO scheduling health checks: 2019-11-04 14:12:16,708 INFO scheduling jobs: 2019-11-04 14:12:16,711 DEBUG - qemu 2019-11-04 14:12:28,030 DEBUG lava-logs => PING(20) 2019-11-04 14:12:35,108 DEBUG Dhanu => PING(20) 2019-11-04 14:12:36,738 INFO scheduling health checks: 2019-11-04 14:12:36,754 INFO scheduling jobs: 2019-11-04 14:12:36,756 DEBUG - qemu 2019-11-04 14:12:48,049 DEBUG lava-logs => PING(20) 2019-11-04 14:12:55,125 DEBUG Dhanu => PING(20) 2019-11-04 14:12:56,777 INFO scheduling health checks: 2019-11-04 14:12:56,800 INFO scheduling jobs: 2019-11-04 14:12:56,803 DEBUG - qemu 2019-11-04 14:13:08,065 DEBUG lava-logs => PING(20) 2019-11-04 14:13:15,147 DEBUG Dhanu => PING(20) 2019-11-04 14:13:16,834 INFO scheduling health checks: 2019-11-04 14:13:16,847 INFO scheduling jobs: 2019-11-04 14:13:16,849 DEBUG - qemu 2019-11-04 14:13:28,083 DEBUG lava-logs => PING(20) 2019-11-04 14:13:35,166 DEBUG Dhanu => PING(20) 2019-11-04 14:13:36,883 INFO scheduling health checks: 2019-11-04 14:13:36,905 INFO scheduling jobs: 2019-11-04 14:13:36,906 DEBUG - qemu 2019-11-04 14:13:48,101 DEBUG lava-logs => PING(20) 2019-11-04 14:13:55,181 DEBUG Dhanu => PING(20) 2019-11-04 14:13:56,987 INFO scheduling health checks: 2019-11-04 14:13:57,006 INFO scheduling jobs: 2019-11-04 14:13:57,008 DEBUG - qemu 2019-11-04 14:14:08,118 DEBUG lava-logs => PING(20) 2019-11-04 14:14:15,202 DEBUG Dhanu => PING(20) 2019-11-04 14:14:17,040 INFO scheduling health checks: 2019-11-04 14:14:17,064 INFO scheduling jobs: 2019-11-04 14:14:17,066 DEBUG - qemu 2019-11-04 14:14:28,137 DEBUG lava-logs => PING(20) 2019-11-04 14:14:35,222 DEBUG Dhanu => PING(20) 2019-11-04 14:14:37,102 INFO scheduling health checks: 2019-11-04 14:14:37,122 INFO scheduling jobs: 2019-11-04 14:14:37,125 DEBUG - qemu 2019-11-04 14:14:48,157 DEBUG lava-logs => PING(20) 2019-11-04 14:14:55,241 DEBUG Dhanu => PING(20) 2019-11-04 14:14:57,159 INFO scheduling health checks: 2019-11-04 14:14:57,180 INFO scheduling jobs: 2019-11-04 14:14:57,181 DEBUG - qemu 2019-11-04 14:15:08,173 DEBUG lava-logs => PING(20) 2019-11-04 14:15:15,257 DEBUG Dhanu => PING(20) 2019-11-04 14:15:17,217 INFO scheduling health checks: 2019-11-04 14:15:17,236 INFO scheduling jobs: 2019-11-04 14:15:17,237 DEBUG - qemu 2019-11-04 14:15:28,220 DEBUG lava-logs => PING(20) 2019-11-04 14:15:35,273 DEBUG Dhanu => PING(20) 2019-11-04 14:15:36,516 INFO scheduling health checks: 2019-11-04 14:15:36,616 INFO scheduling jobs: 2019-11-04 14:15:36,620 DEBUG - qemu 2019-11-04 14:15:36,823 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:15:36,823 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 799, in main_loop jobs = schedule(self.logger, self.events["available_dt"]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:15:36,891 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:15:46,656 INFO [INIT] Dropping privileges 2019-11-04 14:15:46,657 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:15:46,657 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:15:46,660 INFO [INIT] Marking all workers as offline 2019-11-04 14:15:46,696 INFO [INIT] Starting encryption 2019-11-04 14:15:46,697 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:15:46,700 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:15:46,700 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:15:46,718 INFO [INIT] LAVA master has started. 2019-11-04 14:15:46,718 INFO [INIT] Using protocol version 3 2019-11-04 14:15:48,231 DEBUG lava-logs => PING(20) 2019-11-04 14:15:55,294 DEBUG Dhanu => PING(20) 2019-11-04 14:15:55,294 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:16:06,729 INFO scheduling health checks: 2019-11-04 14:16:06,863 INFO scheduling jobs: 2019-11-04 14:16:06,868 DEBUG - qemu 2019-11-04 14:16:06,947 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:16:06,948 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:16:06,953 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:16:08,518 INFO [INIT] Dropping privileges 2019-11-04 14:16:08,518 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:16:08,519 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:16:08,520 INFO [INIT] Marking all workers as offline 2019-11-04 14:16:08,553 INFO [INIT] Starting encryption 2019-11-04 14:16:08,555 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:16:08,556 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:16:08,556 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:16:08,569 INFO [INIT] LAVA master has started. 2019-11-04 14:16:08,569 INFO [INIT] Using protocol version 3 2019-11-04 14:16:08,638 DEBUG lava-logs => PING(20) 2019-11-04 14:16:15,313 DEBUG Dhanu => PING(20) 2019-11-04 14:16:15,313 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:16:28,581 INFO scheduling health checks: 2019-11-04 14:16:28,621 INFO scheduling jobs: 2019-11-04 14:16:28,622 DEBUG - qemu 2019-11-04 14:16:28,697 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:16:28,697 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:16:28,706 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:16:29,997 INFO [INIT] Dropping privileges 2019-11-04 14:16:29,997 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:16:29,997 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:16:29,999 INFO [INIT] Marking all workers as offline 2019-11-04 14:16:30,033 INFO [INIT] Starting encryption 2019-11-04 14:16:30,034 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:16:30,034 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:16:30,035 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:16:30,049 INFO [INIT] LAVA master has started. 2019-11-04 14:16:30,049 INFO [INIT] Using protocol version 3 2019-11-04 14:16:35,325 DEBUG Dhanu => PING(20) 2019-11-04 14:16:35,325 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:16:48,673 DEBUG lava-logs => PING(20) 2019-11-04 14:16:50,050 INFO scheduling health checks: 2019-11-04 14:16:50,090 INFO scheduling jobs: 2019-11-04 14:16:50,091 DEBUG - qemu 2019-11-04 14:16:50,204 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:16:50,204 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:16:50,351 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:16:54,179 INFO [INIT] Dropping privileges 2019-11-04 14:16:54,179 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:16:54,180 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:16:54,183 INFO [INIT] Marking all workers as offline 2019-11-04 14:16:54,225 INFO [INIT] Starting encryption 2019-11-04 14:16:54,226 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:16:54,227 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:16:54,227 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:16:54,242 INFO [INIT] LAVA master has started. 2019-11-04 14:16:54,243 INFO [INIT] Using protocol version 3 2019-11-04 14:16:55,338 DEBUG Dhanu => PING(20) 2019-11-04 14:16:55,339 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:17:08,685 DEBUG lava-logs => PING(20) 2019-11-04 14:17:14,249 INFO scheduling health checks: 2019-11-04 14:17:14,282 INFO scheduling jobs: 2019-11-04 14:17:14,283 DEBUG - qemu 2019-11-04 14:17:14,359 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:17:14,359 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:17:14,364 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:17:15,802 INFO [INIT] Dropping privileges 2019-11-04 14:17:15,802 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:17:15,803 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:17:15,804 INFO [INIT] Marking all workers as offline 2019-11-04 14:17:15,837 INFO [INIT] Starting encryption 2019-11-04 14:17:15,838 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:17:15,839 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:17:15,839 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:17:15,849 INFO [INIT] LAVA master has started. 2019-11-04 14:17:15,850 INFO [INIT] Using protocol version 3 2019-11-04 14:17:15,959 DEBUG Dhanu => PING(20) 2019-11-04 14:17:15,959 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:17:28,698 DEBUG lava-logs => PING(20) 2019-11-04 14:17:35,856 INFO scheduling health checks: 2019-11-04 14:17:35,895 INFO scheduling jobs: 2019-11-04 14:17:35,897 DEBUG - qemu 2019-11-04 14:17:35,970 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:17:35,971 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:17:35,975 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:17:37,279 INFO [INIT] Dropping privileges 2019-11-04 14:17:37,279 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:17:37,279 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:17:37,281 INFO [INIT] Marking all workers as offline 2019-11-04 14:17:37,317 INFO [INIT] Starting encryption 2019-11-04 14:17:37,318 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:17:37,318 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:17:37,318 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:17:37,334 INFO [INIT] LAVA master has started. 2019-11-04 14:17:37,334 INFO [INIT] Using protocol version 3 2019-11-04 14:17:37,530 DEBUG Dhanu => PING(20) 2019-11-04 14:17:37,530 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:17:48,709 DEBUG lava-logs => PING(20) 2019-11-04 14:17:57,340 INFO scheduling health checks: 2019-11-04 14:17:57,386 INFO scheduling jobs: 2019-11-04 14:17:57,387 DEBUG - qemu 2019-11-04 14:17:57,483 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:17:57,483 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:17:57,548 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:18:02,524 INFO [INIT] Dropping privileges 2019-11-04 14:18:02,525 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:18:02,525 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:18:02,528 INFO [INIT] Marking all workers as offline 2019-11-04 14:18:02,564 INFO [INIT] Starting encryption 2019-11-04 14:18:02,565 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:18:02,565 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:18:02,566 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:18:02,578 INFO [INIT] LAVA master has started. 2019-11-04 14:18:02,578 INFO [INIT] Using protocol version 3 2019-11-04 14:18:08,721 DEBUG lava-logs => PING(20) 2019-11-04 14:18:17,561 DEBUG Dhanu => PING(20) 2019-11-04 14:18:17,562 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:18:22,580 INFO scheduling health checks: 2019-11-04 14:18:22,610 INFO scheduling jobs: 2019-11-04 14:18:22,612 DEBUG - qemu 2019-11-04 14:18:22,696 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:18:22,697 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:18:22,703 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:18:24,004 INFO [INIT] Dropping privileges 2019-11-04 14:18:24,005 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:18:24,005 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:18:24,008 INFO [INIT] Marking all workers as offline 2019-11-04 14:18:24,104 INFO [INIT] Starting encryption 2019-11-04 14:18:24,105 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:18:24,105 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:18:24,106 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:18:24,120 INFO [INIT] LAVA master has started. 2019-11-04 14:18:24,120 INFO [INIT] Using protocol version 3 2019-11-04 14:18:28,733 DEBUG lava-logs => PING(20) 2019-11-04 14:18:37,577 DEBUG Dhanu => PING(20) 2019-11-04 14:18:37,577 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:18:44,125 INFO scheduling health checks: 2019-11-04 14:18:44,157 INFO scheduling jobs: 2019-11-04 14:18:44,159 DEBUG - qemu 2019-11-04 14:18:44,236 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:18:44,237 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:18:44,243 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:18:45,479 INFO [INIT] Dropping privileges 2019-11-04 14:18:45,480 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:18:45,480 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:18:45,481 INFO [INIT] Marking all workers as offline 2019-11-04 14:18:45,513 INFO [INIT] Starting encryption 2019-11-04 14:18:45,515 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:18:45,515 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:18:45,515 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:18:45,530 INFO [INIT] LAVA master has started. 2019-11-04 14:18:45,530 INFO [INIT] Using protocol version 3 2019-11-04 14:18:48,743 DEBUG lava-logs => PING(20) 2019-11-04 14:18:57,589 DEBUG Dhanu => PING(20) 2019-11-04 14:18:57,589 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:19:05,536 INFO scheduling health checks: 2019-11-04 14:19:05,570 INFO scheduling jobs: 2019-11-04 14:19:05,573 DEBUG - qemu 2019-11-04 14:19:05,670 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:19:05,670 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:19:05,731 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:19:09,131 INFO [INIT] Dropping privileges 2019-11-04 14:19:09,131 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:19:09,132 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:19:09,134 INFO [INIT] Marking all workers as offline 2019-11-04 14:19:09,171 INFO [INIT] Starting encryption 2019-11-04 14:19:09,172 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:19:09,173 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:19:09,173 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:19:09,201 INFO [INIT] LAVA master has started. 2019-11-04 14:19:09,201 INFO [INIT] Using protocol version 3 2019-11-04 14:19:09,321 DEBUG lava-logs => PING(20) 2019-11-04 14:19:17,605 DEBUG Dhanu => PING(20) 2019-11-04 14:19:17,605 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:19:29,213 INFO scheduling health checks: 2019-11-04 14:19:29,250 INFO scheduling jobs: 2019-11-04 14:19:29,252 DEBUG - qemu 2019-11-04 14:19:29,348 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:19:29,348 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:19:29,411 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:19:32,504 INFO [INIT] Dropping privileges 2019-11-04 14:19:32,504 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:19:32,504 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:19:32,506 INFO [INIT] Marking all workers as offline 2019-11-04 14:19:32,584 INFO [INIT] Starting encryption 2019-11-04 14:19:32,585 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:19:32,586 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:19:32,586 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:19:32,602 INFO [INIT] LAVA master has started. 2019-11-04 14:19:32,602 INFO [INIT] Using protocol version 3 2019-11-04 14:19:37,627 DEBUG Dhanu => PING(20) 2019-11-04 14:19:37,627 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:19:49,349 DEBUG lava-logs => PING(20) 2019-11-04 14:19:52,604 INFO scheduling health checks: 2019-11-04 14:19:52,631 INFO scheduling jobs: 2019-11-04 14:19:52,633 DEBUG - qemu 2019-11-04 14:19:52,713 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:19:52,713 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:19:52,718 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:19:53,989 INFO [INIT] Dropping privileges 2019-11-04 14:19:53,989 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:19:53,990 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:19:53,991 INFO [INIT] Marking all workers as offline 2019-11-04 14:19:54,031 INFO [INIT] Starting encryption 2019-11-04 14:19:54,033 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:19:54,033 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:19:54,034 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:19:54,044 INFO [INIT] LAVA master has started. 2019-11-04 14:19:54,045 INFO [INIT] Using protocol version 3 2019-11-04 14:19:57,645 DEBUG Dhanu => PING(20) 2019-11-04 14:19:57,645 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:20:09,361 DEBUG lava-logs => PING(20) 2019-11-04 14:20:14,049 INFO scheduling health checks: 2019-11-04 14:20:14,088 INFO scheduling jobs: 2019-11-04 14:20:14,090 DEBUG - qemu 2019-11-04 14:20:14,173 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:20:14,174 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:20:14,178 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:20:17,969 INFO [INIT] Dropping privileges 2019-11-04 14:20:17,970 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:20:17,970 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:20:17,972 INFO [INIT] Marking all workers as offline 2019-11-04 14:20:18,000 INFO [INIT] Starting encryption 2019-11-04 14:20:18,000 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:20:18,001 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:20:18,001 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:20:18,011 INFO [INIT] LAVA master has started. 2019-11-04 14:20:18,011 INFO [INIT] Using protocol version 3 2019-11-04 14:20:18,120 DEBUG Dhanu => PING(20) 2019-11-04 14:20:18,121 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:20:29,374 DEBUG lava-logs => PING(20) 2019-11-04 14:20:38,020 INFO scheduling health checks: 2019-11-04 14:20:38,079 INFO scheduling jobs: 2019-11-04 14:20:38,082 DEBUG - qemu 2019-11-04 14:20:38,154 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:20:38,154 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:20:38,158 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:20:41,541 INFO [INIT] Dropping privileges 2019-11-04 14:20:41,541 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:20:41,541 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:20:41,543 INFO [INIT] Marking all workers as offline 2019-11-04 14:20:41,580 INFO [INIT] Starting encryption 2019-11-04 14:20:41,582 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:20:41,582 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:20:41,582 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:20:41,592 INFO [INIT] LAVA master has started. 2019-11-04 14:20:41,592 INFO [INIT] Using protocol version 3 2019-11-04 14:20:49,393 DEBUG lava-logs => PING(20) 2019-11-04 14:20:58,153 DEBUG Dhanu => PING(20) 2019-11-04 14:20:58,153 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:21:01,596 INFO scheduling health checks: 2019-11-04 14:21:01,626 INFO scheduling jobs: 2019-11-04 14:21:01,628 DEBUG - qemu 2019-11-04 14:21:01,702 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:21:01,702 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:21:01,707 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:21:02,982 INFO [INIT] Dropping privileges 2019-11-04 14:21:02,982 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:21:02,983 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:21:02,985 INFO [INIT] Marking all workers as offline 2019-11-04 14:21:03,027 INFO [INIT] Starting encryption 2019-11-04 14:21:03,029 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:21:03,029 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:21:03,029 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:21:03,039 INFO [INIT] LAVA master has started. 2019-11-04 14:21:03,039 INFO [INIT] Using protocol version 3 2019-11-04 14:21:09,405 DEBUG lava-logs => PING(20) 2019-11-04 14:21:18,167 DEBUG Dhanu => PING(20) 2019-11-04 14:21:18,167 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:21:23,040 INFO scheduling health checks: 2019-11-04 14:21:23,106 INFO scheduling jobs: 2019-11-04 14:21:23,108 DEBUG - qemu 2019-11-04 14:21:23,189 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:21:23,189 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:21:23,197 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:21:27,008 INFO [INIT] Dropping privileges 2019-11-04 14:21:27,009 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:21:27,009 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:21:27,011 INFO [INIT] Marking all workers as offline 2019-11-04 14:21:27,079 INFO [INIT] Starting encryption 2019-11-04 14:21:27,080 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:21:27,081 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:21:27,081 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:21:27,092 INFO [INIT] LAVA master has started. 2019-11-04 14:21:27,092 INFO [INIT] Using protocol version 3 2019-11-04 14:21:29,417 DEBUG lava-logs => PING(20) 2019-11-04 14:21:38,182 DEBUG Dhanu => PING(20) 2019-11-04 14:21:38,183 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:21:47,100 INFO scheduling health checks: 2019-11-04 14:21:47,153 INFO scheduling jobs: 2019-11-04 14:21:47,156 DEBUG - qemu 2019-11-04 14:21:47,255 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:21:47,255 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:21:47,315 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:21:51,029 INFO [INIT] Dropping privileges 2019-11-04 14:21:51,029 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:21:51,029 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:21:51,031 INFO [INIT] Marking all workers as offline 2019-11-04 14:21:51,065 INFO [INIT] Starting encryption 2019-11-04 14:21:51,066 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:21:51,066 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:21:51,067 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:21:51,081 INFO [INIT] LAVA master has started. 2019-11-04 14:21:51,081 INFO [INIT] Using protocol version 3 2019-11-04 14:21:51,223 DEBUG lava-logs => PING(20) 2019-11-04 14:21:58,201 DEBUG Dhanu => PING(20) 2019-11-04 14:21:58,201 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:22:11,083 INFO scheduling health checks: 2019-11-04 14:22:11,111 INFO scheduling jobs: 2019-11-04 14:22:11,113 DEBUG - qemu 2019-11-04 14:22:11,181 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:22:11,181 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:22:11,186 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:22:12,474 INFO [INIT] Dropping privileges 2019-11-04 14:22:12,474 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:22:12,475 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:22:12,476 INFO [INIT] Marking all workers as offline 2019-11-04 14:22:12,501 INFO [INIT] Starting encryption 2019-11-04 14:22:12,502 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:22:12,502 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:22:12,503 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:22:12,518 INFO [INIT] LAVA master has started. 2019-11-04 14:22:12,519 INFO [INIT] Using protocol version 3 2019-11-04 14:22:12,632 DEBUG lava-logs => PING(20) 2019-11-04 14:22:18,216 DEBUG Dhanu => PING(20) 2019-11-04 14:22:18,216 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:22:32,533 INFO scheduling health checks: 2019-11-04 14:22:32,568 INFO scheduling jobs: 2019-11-04 14:22:32,570 DEBUG - qemu 2019-11-04 14:22:32,652 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:22:32,652 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:22:32,660 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:22:35,088 INFO [INIT] Dropping privileges 2019-11-04 14:22:35,107 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:22:35,107 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:22:35,113 INFO [INIT] Marking all workers as offline 2019-11-04 14:22:35,147 INFO [INIT] Starting encryption 2019-11-04 14:22:35,149 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:22:35,149 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:22:35,150 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:22:35,188 INFO [INIT] LAVA master has started. 2019-11-04 14:22:35,188 INFO [INIT] Using protocol version 3 2019-11-04 14:22:38,235 DEBUG Dhanu => PING(20) 2019-11-04 14:22:38,236 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:22:52,667 DEBUG lava-logs => PING(20) 2019-11-04 14:22:55,190 INFO scheduling health checks: 2019-11-04 14:22:55,215 INFO scheduling jobs: 2019-11-04 14:22:55,217 DEBUG - qemu 2019-11-04 14:22:55,294 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:22:55,294 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:22:55,343 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:23:00,402 INFO [INIT] Dropping privileges 2019-11-04 14:23:00,402 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:23:00,403 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:23:00,404 INFO [INIT] Marking all workers as offline 2019-11-04 14:23:00,438 INFO [INIT] Starting encryption 2019-11-04 14:23:00,439 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:23:00,440 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:23:00,440 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:23:00,455 INFO [INIT] LAVA master has started. 2019-11-04 14:23:00,456 INFO [INIT] Using protocol version 3 2019-11-04 14:23:00,495 DEBUG Dhanu => PING(20) 2019-11-04 14:23:00,496 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:23:12,681 DEBUG lava-logs => PING(20) 2019-11-04 14:23:20,464 INFO scheduling health checks: 2019-11-04 14:23:20,495 INFO scheduling jobs: 2019-11-04 14:23:20,497 DEBUG - qemu 2019-11-04 14:23:20,575 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:23:20,575 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:23:20,585 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:23:21,997 INFO [INIT] Dropping privileges 2019-11-04 14:23:21,998 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:23:21,998 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:23:22,000 INFO [INIT] Marking all workers as offline 2019-11-04 14:23:22,040 INFO [INIT] Starting encryption 2019-11-04 14:23:22,041 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:23:22,041 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:23:22,042 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:23:22,054 INFO [INIT] LAVA master has started. 2019-11-04 14:23:22,054 INFO [INIT] Using protocol version 3 2019-11-04 14:23:32,694 DEBUG lava-logs => PING(20) 2019-11-04 14:23:40,529 DEBUG Dhanu => PING(20) 2019-11-04 14:23:40,529 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:23:42,055 INFO scheduling health checks: 2019-11-04 14:23:42,098 INFO scheduling jobs: 2019-11-04 14:23:42,100 DEBUG - qemu 2019-11-04 14:23:42,180 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:23:42,180 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:23:42,185 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:23:43,506 INFO [INIT] Dropping privileges 2019-11-04 14:23:43,506 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:23:43,506 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:23:43,508 INFO [INIT] Marking all workers as offline 2019-11-04 14:23:43,542 INFO [INIT] Starting encryption 2019-11-04 14:23:43,543 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:23:43,544 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:23:43,544 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:23:43,559 INFO [INIT] LAVA master has started. 2019-11-04 14:23:43,559 INFO [INIT] Using protocol version 3 2019-11-04 14:23:52,715 DEBUG lava-logs => PING(20) 2019-11-04 14:24:00,547 DEBUG Dhanu => PING(20) 2019-11-04 14:24:00,547 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:24:03,561 INFO scheduling health checks: 2019-11-04 14:24:03,599 INFO scheduling jobs: 2019-11-04 14:24:03,601 DEBUG - qemu 2019-11-04 14:24:03,674 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:24:03,674 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:24:03,728 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:24:09,710 INFO [INIT] Dropping privileges 2019-11-04 14:24:09,710 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:24:09,710 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:24:09,712 INFO [INIT] Marking all workers as offline 2019-11-04 14:24:09,745 INFO [INIT] Starting encryption 2019-11-04 14:24:09,746 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:24:09,746 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:24:09,746 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:24:09,755 INFO [INIT] LAVA master has started. 2019-11-04 14:24:09,756 INFO [INIT] Using protocol version 3 2019-11-04 14:24:12,728 DEBUG lava-logs => PING(20) 2019-11-04 14:24:20,561 DEBUG Dhanu => PING(20) 2019-11-04 14:24:20,561 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:24:29,764 INFO scheduling health checks: 2019-11-04 14:24:29,802 INFO scheduling jobs: 2019-11-04 14:24:29,804 DEBUG - qemu 2019-11-04 14:24:29,883 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:24:29,883 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:24:29,896 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:24:31,228 INFO [INIT] Dropping privileges 2019-11-04 14:24:31,228 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:24:31,229 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:24:31,230 INFO [INIT] Marking all workers as offline 2019-11-04 14:24:31,259 INFO [INIT] Starting encryption 2019-11-04 14:24:31,260 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:24:31,261 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:24:31,261 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:24:31,273 INFO [INIT] LAVA master has started. 2019-11-04 14:24:31,274 INFO [INIT] Using protocol version 3 2019-11-04 14:24:32,737 DEBUG lava-logs => PING(20) 2019-11-04 14:24:40,577 DEBUG Dhanu => PING(20) 2019-11-04 14:24:40,578 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:24:51,284 INFO scheduling health checks: 2019-11-04 14:24:51,327 INFO scheduling jobs: 2019-11-04 14:24:51,329 DEBUG - qemu 2019-11-04 14:24:51,411 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:24:51,411 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:24:51,416 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:24:52,721 INFO [INIT] Dropping privileges 2019-11-04 14:24:52,722 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:24:52,722 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:24:52,723 INFO [INIT] Marking all workers as offline 2019-11-04 14:24:52,749 INFO [INIT] Starting encryption 2019-11-04 14:24:52,750 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:24:52,751 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:24:52,751 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:24:52,765 INFO [INIT] LAVA master has started. 2019-11-04 14:24:52,766 INFO [INIT] Using protocol version 3 2019-11-04 14:24:52,776 DEBUG lava-logs => PING(20) 2019-11-04 14:25:00,597 DEBUG Dhanu => PING(20) 2019-11-04 14:25:00,598 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:25:12,777 INFO scheduling health checks: 2019-11-04 14:25:12,851 INFO scheduling jobs: 2019-11-04 14:25:12,852 DEBUG - qemu 2019-11-04 14:25:12,942 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:25:12,942 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:25:13,001 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:25:18,071 INFO [INIT] Dropping privileges 2019-11-04 14:25:18,072 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:25:18,072 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:25:18,075 INFO [INIT] Marking all workers as offline 2019-11-04 14:25:18,108 INFO [INIT] Starting encryption 2019-11-04 14:25:18,109 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:25:18,109 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:25:18,109 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:25:18,124 INFO [INIT] LAVA master has started. 2019-11-04 14:25:18,125 INFO [INIT] Using protocol version 3 2019-11-04 14:25:20,609 DEBUG Dhanu => PING(20) 2019-11-04 14:25:20,610 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:25:32,806 DEBUG lava-logs => PING(20) 2019-11-04 14:25:38,128 INFO scheduling health checks: 2019-11-04 14:25:38,176 INFO scheduling jobs: 2019-11-04 14:25:38,178 DEBUG - qemu 2019-11-04 14:25:38,257 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:25:38,257 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:25:38,264 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:25:39,483 INFO [INIT] Dropping privileges 2019-11-04 14:25:39,483 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:25:39,483 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:25:39,485 INFO [INIT] Marking all workers as offline 2019-11-04 14:25:39,520 INFO [INIT] Starting encryption 2019-11-04 14:25:39,521 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:25:39,522 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:25:39,522 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:25:39,534 INFO [INIT] LAVA master has started. 2019-11-04 14:25:39,535 INFO [INIT] Using protocol version 3 2019-11-04 14:25:40,623 DEBUG Dhanu => PING(20) 2019-11-04 14:25:40,623 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:25:52,821 DEBUG lava-logs => PING(20) 2019-11-04 14:25:59,540 INFO scheduling health checks: 2019-11-04 14:25:59,577 INFO scheduling jobs: 2019-11-04 14:25:59,579 DEBUG - qemu 2019-11-04 14:25:59,659 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:25:59,659 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:25:59,664 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:26:00,945 INFO [INIT] Dropping privileges 2019-11-04 14:26:00,945 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:26:00,945 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:26:00,947 INFO [INIT] Marking all workers as offline 2019-11-04 14:26:00,990 INFO [INIT] Starting encryption 2019-11-04 14:26:00,991 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:26:00,991 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:26:00,992 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:26:01,004 INFO [INIT] LAVA master has started. 2019-11-04 14:26:01,005 INFO [INIT] Using protocol version 3 2019-11-04 14:26:01,039 DEBUG Dhanu => PING(20) 2019-11-04 14:26:01,040 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:26:12,834 DEBUG lava-logs => PING(20) 2019-11-04 14:26:21,013 INFO scheduling health checks: 2019-11-04 14:26:21,080 INFO scheduling jobs: 2019-11-04 14:26:21,083 DEBUG - qemu 2019-11-04 14:26:21,194 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:26:21,194 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:26:21,253 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:26:26,951 INFO [INIT] Dropping privileges 2019-11-04 14:26:26,952 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:26:26,952 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:26:26,954 INFO [INIT] Marking all workers as offline 2019-11-04 14:26:26,981 INFO [INIT] Starting encryption 2019-11-04 14:26:26,982 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:26:26,983 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:26:26,983 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:26:26,995 INFO [INIT] LAVA master has started. 2019-11-04 14:26:26,995 INFO [INIT] Using protocol version 3 2019-11-04 14:26:32,850 DEBUG lava-logs => PING(20) 2019-11-04 14:26:41,069 DEBUG Dhanu => PING(20) 2019-11-04 14:26:41,069 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:26:47,000 INFO scheduling health checks: 2019-11-04 14:26:47,038 INFO scheduling jobs: 2019-11-04 14:26:47,040 DEBUG - qemu 2019-11-04 14:26:47,123 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:26:47,123 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:26:47,131 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:26:48,468 INFO [INIT] Dropping privileges 2019-11-04 14:26:48,469 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:26:48,469 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:26:48,472 INFO [INIT] Marking all workers as offline 2019-11-04 14:26:48,505 INFO [INIT] Starting encryption 2019-11-04 14:26:48,506 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:26:48,506 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:26:48,506 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:26:48,518 INFO [INIT] LAVA master has started. 2019-11-04 14:26:48,519 INFO [INIT] Using protocol version 3 2019-11-04 14:26:52,866 DEBUG lava-logs => PING(20) 2019-11-04 14:27:01,085 DEBUG Dhanu => PING(20) 2019-11-04 14:27:01,085 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:27:08,527 INFO scheduling health checks: 2019-11-04 14:27:08,562 INFO scheduling jobs: 2019-11-04 14:27:08,564 DEBUG - qemu 2019-11-04 14:27:08,633 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:27:08,634 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:27:08,645 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:27:10,001 INFO [INIT] Dropping privileges 2019-11-04 14:27:10,001 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:27:10,002 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:27:10,003 INFO [INIT] Marking all workers as offline 2019-11-04 14:27:10,041 INFO [INIT] Starting encryption 2019-11-04 14:27:10,042 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:27:10,042 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:27:10,042 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:27:10,055 INFO [INIT] LAVA master has started. 2019-11-04 14:27:10,055 INFO [INIT] Using protocol version 3 2019-11-04 14:27:12,881 DEBUG lava-logs => PING(20) 2019-11-04 14:27:21,097 DEBUG Dhanu => PING(20) 2019-11-04 14:27:21,098 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:27:30,064 INFO scheduling health checks: 2019-11-04 14:27:30,120 INFO scheduling jobs: 2019-11-04 14:27:30,121 DEBUG - qemu 2019-11-04 14:27:30,221 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:27:30,221 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:27:30,283 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:27:35,813 INFO [INIT] Dropping privileges 2019-11-04 14:27:35,813 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:27:35,814 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:27:35,815 INFO [INIT] Marking all workers as offline 2019-11-04 14:27:35,844 INFO [INIT] Starting encryption 2019-11-04 14:27:35,846 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:27:35,846 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:27:35,846 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:27:35,861 INFO [INIT] LAVA master has started. 2019-11-04 14:27:35,862 INFO [INIT] Using protocol version 3 2019-11-04 14:27:35,947 DEBUG lava-logs => PING(20) 2019-11-04 14:27:41,109 DEBUG Dhanu => PING(20) 2019-11-04 14:27:41,110 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:27:55,872 INFO scheduling health checks: 2019-11-04 14:27:55,911 INFO scheduling jobs: 2019-11-04 14:27:55,914 DEBUG - qemu 2019-11-04 14:27:55,993 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:27:55,994 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:27:55,998 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:27:57,233 INFO [INIT] Dropping privileges 2019-11-04 14:27:57,233 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:27:57,233 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:27:57,235 INFO [INIT] Marking all workers as offline 2019-11-04 14:27:57,312 INFO [INIT] Starting encryption 2019-11-04 14:27:57,313 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:27:57,313 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:27:57,313 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:27:57,325 INFO [INIT] LAVA master has started. 2019-11-04 14:27:57,326 INFO [INIT] Using protocol version 3 2019-11-04 14:28:01,130 DEBUG Dhanu => PING(20) 2019-11-04 14:28:01,130 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:28:15,983 DEBUG lava-logs => PING(20) 2019-11-04 14:28:17,327 INFO scheduling health checks: 2019-11-04 14:28:17,358 INFO scheduling jobs: 2019-11-04 14:28:17,359 DEBUG - qemu 2019-11-04 14:28:17,430 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:28:17,430 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:28:17,435 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:28:18,765 INFO [INIT] Dropping privileges 2019-11-04 14:28:18,765 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:28:18,766 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:28:18,768 INFO [INIT] Marking all workers as offline 2019-11-04 14:28:18,803 INFO [INIT] Starting encryption 2019-11-04 14:28:18,804 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:28:18,804 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:28:18,805 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:28:18,818 INFO [INIT] LAVA master has started. 2019-11-04 14:28:18,818 INFO [INIT] Using protocol version 3 2019-11-04 14:28:21,149 DEBUG Dhanu => PING(20) 2019-11-04 14:28:21,149 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:28:36,000 DEBUG lava-logs => PING(20) 2019-11-04 14:28:38,820 INFO scheduling health checks: 2019-11-04 14:28:38,882 INFO scheduling jobs: 2019-11-04 14:28:38,885 DEBUG - qemu 2019-11-04 14:28:38,979 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:28:38,979 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:28:39,043 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:28:43,152 INFO [INIT] Dropping privileges 2019-11-04 14:28:43,176 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:28:43,177 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:28:43,180 INFO [INIT] Marking all workers as offline 2019-11-04 14:28:43,217 INFO [INIT] Starting encryption 2019-11-04 14:28:43,219 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:28:43,219 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:28:43,220 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:28:43,241 INFO [INIT] LAVA master has started. 2019-11-04 14:28:43,241 INFO [INIT] Using protocol version 3 2019-11-04 14:28:43,322 DEBUG Dhanu => PING(20) 2019-11-04 14:28:43,323 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:28:56,013 DEBUG lava-logs => PING(20) 2019-11-04 14:29:03,247 INFO scheduling health checks: 2019-11-04 14:29:03,307 INFO scheduling jobs: 2019-11-04 14:29:03,310 DEBUG - qemu 2019-11-04 14:29:03,404 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:29:03,404 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:29:03,471 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:29:06,376 INFO [INIT] Dropping privileges 2019-11-04 14:29:06,377 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:29:06,377 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:29:06,379 INFO [INIT] Marking all workers as offline 2019-11-04 14:29:06,408 INFO [INIT] Starting encryption 2019-11-04 14:29:06,409 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:29:06,409 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:29:06,409 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:29:06,421 INFO [INIT] LAVA master has started. 2019-11-04 14:29:06,421 INFO [INIT] Using protocol version 3 2019-11-04 14:29:16,035 DEBUG lava-logs => PING(20) 2019-11-04 14:29:23,357 DEBUG Dhanu => PING(20) 2019-11-04 14:29:23,358 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:29:26,424 INFO scheduling health checks: 2019-11-04 14:29:26,453 INFO scheduling jobs: 2019-11-04 14:29:26,455 DEBUG - qemu 2019-11-04 14:29:26,529 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:29:26,530 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:29:26,535 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:29:27,782 INFO [INIT] Dropping privileges 2019-11-04 14:29:27,782 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:29:27,782 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:29:27,785 INFO [INIT] Marking all workers as offline 2019-11-04 14:29:27,821 INFO [INIT] Starting encryption 2019-11-04 14:29:27,822 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:29:27,822 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:29:27,822 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:29:27,838 INFO [INIT] LAVA master has started. 2019-11-04 14:29:27,838 INFO [INIT] Using protocol version 3 2019-11-04 14:29:36,053 DEBUG lava-logs => PING(20) 2019-11-04 14:29:43,373 DEBUG Dhanu => PING(20) 2019-11-04 14:29:43,374 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:29:47,840 INFO scheduling health checks: 2019-11-04 14:29:47,878 INFO scheduling jobs: 2019-11-04 14:29:47,880 DEBUG - qemu 2019-11-04 14:29:47,958 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:29:47,958 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:29:47,963 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:29:52,295 INFO [INIT] Dropping privileges 2019-11-04 14:29:52,296 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:29:52,296 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:29:52,298 INFO [INIT] Marking all workers as offline 2019-11-04 14:29:52,323 INFO [INIT] Starting encryption 2019-11-04 14:29:52,324 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:29:52,325 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:29:52,325 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:29:52,336 INFO [INIT] LAVA master has started. 2019-11-04 14:29:52,336 INFO [INIT] Using protocol version 3 2019-11-04 14:29:56,065 DEBUG lava-logs => PING(20) 2019-11-04 14:30:03,389 DEBUG Dhanu => PING(20) 2019-11-04 14:30:03,389 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:30:12,344 INFO scheduling health checks: 2019-11-04 14:30:12,369 INFO scheduling jobs: 2019-11-04 14:30:12,370 DEBUG - qemu 2019-11-04 14:30:12,440 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:30:12,441 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:30:12,445 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:30:15,363 INFO [INIT] Dropping privileges 2019-11-04 14:30:15,364 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:30:15,364 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:30:15,366 INFO [INIT] Marking all workers as offline 2019-11-04 14:30:15,392 INFO [INIT] Starting encryption 2019-11-04 14:30:15,393 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:30:15,394 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:30:15,394 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:30:15,408 INFO [INIT] LAVA master has started. 2019-11-04 14:30:15,409 INFO [INIT] Using protocol version 3 2019-11-04 14:30:16,078 DEBUG lava-logs => PING(20) 2019-11-04 14:30:23,405 DEBUG Dhanu => PING(20) 2019-11-04 14:30:23,406 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:30:35,420 INFO scheduling health checks: 2019-11-04 14:30:35,449 INFO scheduling jobs: 2019-11-04 14:30:35,451 DEBUG - qemu 2019-11-04 14:30:35,525 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:30:35,525 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:30:35,529 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:30:36,785 INFO [INIT] Dropping privileges 2019-11-04 14:30:36,785 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:30:36,785 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:30:36,787 INFO [INIT] Marking all workers as offline 2019-11-04 14:30:36,816 INFO [INIT] Starting encryption 2019-11-04 14:30:36,817 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:30:36,817 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:30:36,817 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:30:36,827 INFO [INIT] LAVA master has started. 2019-11-04 14:30:36,828 INFO [INIT] Using protocol version 3 2019-11-04 14:30:36,903 DEBUG lava-logs => PING(20) 2019-11-04 14:30:43,416 DEBUG Dhanu => PING(20) 2019-11-04 14:30:43,417 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:30:56,840 INFO scheduling health checks: 2019-11-04 14:30:56,884 INFO scheduling jobs: 2019-11-04 14:30:56,886 DEBUG - qemu 2019-11-04 14:30:56,960 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:30:56,961 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:30:56,965 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:31:01,161 INFO [INIT] Dropping privileges 2019-11-04 14:31:01,161 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:31:01,162 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:31:01,163 INFO [INIT] Marking all workers as offline 2019-11-04 14:31:01,197 INFO [INIT] Starting encryption 2019-11-04 14:31:01,198 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:31:01,199 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:31:01,199 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:31:01,217 INFO [INIT] LAVA master has started. 2019-11-04 14:31:01,217 INFO [INIT] Using protocol version 3 2019-11-04 14:31:03,432 DEBUG Dhanu => PING(20) 2019-11-04 14:31:03,433 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:31:16,933 DEBUG lava-logs => PING(20) 2019-11-04 14:31:21,220 INFO scheduling health checks: 2019-11-04 14:31:21,287 INFO scheduling jobs: 2019-11-04 14:31:21,289 DEBUG - qemu 2019-11-04 14:31:21,392 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:31:21,392 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:31:21,450 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:31:24,787 INFO [INIT] Dropping privileges 2019-11-04 14:31:24,788 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:31:24,788 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:31:24,790 INFO [INIT] Marking all workers as offline 2019-11-04 14:31:24,832 INFO [INIT] Starting encryption 2019-11-04 14:31:24,833 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:31:24,834 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:31:24,834 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:31:24,846 INFO [INIT] LAVA master has started. 2019-11-04 14:31:24,847 INFO [INIT] Using protocol version 3 2019-11-04 14:31:24,864 DEBUG Dhanu => PING(20) 2019-11-04 14:31:24,865 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:31:36,945 DEBUG lava-logs => PING(20) 2019-11-04 14:31:44,852 INFO scheduling health checks: 2019-11-04 14:31:44,889 INFO scheduling jobs: 2019-11-04 14:31:44,891 DEBUG - qemu 2019-11-04 14:31:44,971 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:31:44,971 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:31:44,975 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:31:46,228 INFO [INIT] Dropping privileges 2019-11-04 14:31:46,228 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:31:46,228 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:31:46,230 INFO [INIT] Marking all workers as offline 2019-11-04 14:31:46,256 INFO [INIT] Starting encryption 2019-11-04 14:31:46,257 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:31:46,258 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:31:46,258 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:31:46,271 INFO [INIT] LAVA master has started. 2019-11-04 14:31:46,272 INFO [INIT] Using protocol version 3 2019-11-04 14:31:56,953 DEBUG lava-logs => PING(20) 2019-11-04 14:32:04,901 DEBUG Dhanu => PING(20) 2019-11-04 14:32:04,902 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:32:06,272 INFO scheduling health checks: 2019-11-04 14:32:06,291 INFO scheduling jobs: 2019-11-04 14:32:06,293 DEBUG - qemu 2019-11-04 14:32:06,366 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:32:06,366 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:32:06,372 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:32:10,973 INFO [INIT] Dropping privileges 2019-11-04 14:32:10,986 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:32:10,986 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:32:10,990 INFO [INIT] Marking all workers as offline 2019-11-04 14:32:11,026 INFO [INIT] Starting encryption 2019-11-04 14:32:11,027 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:32:11,028 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:32:11,028 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:32:11,041 INFO [INIT] LAVA master has started. 2019-11-04 14:32:11,041 INFO [INIT] Using protocol version 3 2019-11-04 14:32:16,965 DEBUG lava-logs => PING(20) 2019-11-04 14:32:24,916 DEBUG Dhanu => PING(20) 2019-11-04 14:32:24,917 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:32:31,047 INFO scheduling health checks: 2019-11-04 14:32:31,093 INFO scheduling jobs: 2019-11-04 14:32:31,095 DEBUG - qemu 2019-11-04 14:32:31,176 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:32:31,176 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:32:31,184 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:32:34,390 INFO [INIT] Dropping privileges 2019-11-04 14:32:34,391 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:32:34,391 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:32:34,393 INFO [INIT] Marking all workers as offline 2019-11-04 14:32:34,428 INFO [INIT] Starting encryption 2019-11-04 14:32:34,429 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:32:34,430 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:32:34,430 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:32:34,439 INFO [INIT] LAVA master has started. 2019-11-04 14:32:34,440 INFO [INIT] Using protocol version 3 2019-11-04 14:32:36,980 DEBUG lava-logs => PING(20) 2019-11-04 14:32:44,932 DEBUG Dhanu => PING(20) 2019-11-04 14:32:44,933 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:32:54,440 INFO scheduling health checks: 2019-11-04 14:32:54,474 INFO scheduling jobs: 2019-11-04 14:32:54,475 DEBUG - qemu 2019-11-04 14:32:54,551 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:32:54,551 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:32:54,556 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:32:55,745 INFO [INIT] Dropping privileges 2019-11-04 14:32:55,745 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:32:55,745 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:32:55,747 INFO [INIT] Marking all workers as offline 2019-11-04 14:32:55,786 INFO [INIT] Starting encryption 2019-11-04 14:32:55,787 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:32:55,787 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:32:55,788 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:32:55,800 INFO [INIT] LAVA master has started. 2019-11-04 14:32:55,801 INFO [INIT] Using protocol version 3 2019-11-04 14:32:56,993 DEBUG lava-logs => PING(20) 2019-11-04 14:33:04,950 DEBUG Dhanu => PING(20) 2019-11-04 14:33:04,950 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:33:15,808 INFO scheduling health checks: 2019-11-04 14:33:15,842 INFO scheduling jobs: 2019-11-04 14:33:15,845 DEBUG - qemu 2019-11-04 14:33:15,925 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:33:15,926 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:33:15,934 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:33:17,257 INFO [INIT] Dropping privileges 2019-11-04 14:33:17,257 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:33:17,258 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:33:17,259 INFO [INIT] Marking all workers as offline 2019-11-04 14:33:17,298 INFO [INIT] Starting encryption 2019-11-04 14:33:17,299 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:33:17,300 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:33:17,300 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:33:17,309 INFO [INIT] LAVA master has started. 2019-11-04 14:33:17,310 INFO [INIT] Using protocol version 3 2019-11-04 14:33:17,380 DEBUG lava-logs => PING(20) 2019-11-04 14:33:24,969 DEBUG Dhanu => PING(20) 2019-11-04 14:33:24,969 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:33:37,320 INFO scheduling health checks: 2019-11-04 14:33:37,356 INFO scheduling jobs: 2019-11-04 14:33:37,358 DEBUG - qemu 2019-11-04 14:33:37,433 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:33:37,433 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:33:37,442 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:33:43,592 INFO [INIT] Dropping privileges 2019-11-04 14:33:43,593 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:33:43,593 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:33:43,595 INFO [INIT] Marking all workers as offline 2019-11-04 14:33:43,634 INFO [INIT] Starting encryption 2019-11-04 14:33:43,635 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:33:43,635 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:33:43,635 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:33:43,648 INFO [INIT] LAVA master has started. 2019-11-04 14:33:43,649 INFO [INIT] Using protocol version 3 2019-11-04 14:33:44,983 DEBUG Dhanu => PING(20) 2019-11-04 14:33:44,983 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:33:57,415 DEBUG lava-logs => PING(20) 2019-11-04 14:34:03,652 INFO scheduling health checks: 2019-11-04 14:34:03,680 INFO scheduling jobs: 2019-11-04 14:34:03,683 DEBUG - qemu 2019-11-04 14:34:03,756 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:34:03,756 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:34:03,764 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:34:04,993 INFO [INIT] Dropping privileges 2019-11-04 14:34:04,994 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:34:04,994 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:34:04,996 INFO [INIT] Marking all workers as offline 2019-11-04 14:34:05,048 INFO [INIT] Starting encryption 2019-11-04 14:34:05,049 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:34:05,050 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:34:05,050 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:34:05,064 INFO [INIT] LAVA master has started. 2019-11-04 14:34:05,064 INFO [INIT] Using protocol version 3 2019-11-04 14:34:05,153 DEBUG Dhanu => PING(20) 2019-11-04 14:34:05,154 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:34:17,430 DEBUG lava-logs => PING(20) 2019-11-04 14:34:25,068 INFO scheduling health checks: 2019-11-04 14:34:25,105 INFO scheduling jobs: 2019-11-04 14:34:25,106 DEBUG - qemu 2019-11-04 14:34:25,185 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:34:25,186 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:34:25,190 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:34:26,497 INFO [INIT] Dropping privileges 2019-11-04 14:34:26,497 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:34:26,497 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:34:26,499 INFO [INIT] Marking all workers as offline 2019-11-04 14:34:26,550 INFO [INIT] Starting encryption 2019-11-04 14:34:26,552 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:34:26,552 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:34:26,553 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:34:26,566 INFO [INIT] LAVA master has started. 2019-11-04 14:34:26,566 INFO [INIT] Using protocol version 3 2019-11-04 14:34:37,441 DEBUG lava-logs => PING(20) 2019-11-04 14:34:45,189 DEBUG Dhanu => PING(20) 2019-11-04 14:34:45,189 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:34:46,567 INFO scheduling health checks: 2019-11-04 14:34:46,640 INFO scheduling jobs: 2019-11-04 14:34:46,642 DEBUG - qemu 2019-11-04 14:34:46,742 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:34:46,743 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:34:46,803 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:34:53,287 INFO [INIT] Dropping privileges 2019-11-04 14:34:53,288 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:34:53,288 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:34:53,291 INFO [INIT] Marking all workers as offline 2019-11-04 14:34:53,319 INFO [INIT] Starting encryption 2019-11-04 14:34:53,320 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:34:53,321 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:34:53,321 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:34:53,336 INFO [INIT] LAVA master has started. 2019-11-04 14:34:53,336 INFO [INIT] Using protocol version 3 2019-11-04 14:34:57,458 DEBUG lava-logs => PING(20) 2019-11-04 14:35:05,201 DEBUG Dhanu => PING(20) 2019-11-04 14:35:05,201 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:35:13,344 INFO scheduling health checks: 2019-11-04 14:35:13,376 INFO scheduling jobs: 2019-11-04 14:35:13,379 DEBUG - qemu 2019-11-04 14:35:13,454 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:35:13,454 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:35:13,462 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:35:14,732 INFO [INIT] Dropping privileges 2019-11-04 14:35:14,732 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:35:14,733 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:35:14,735 INFO [INIT] Marking all workers as offline 2019-11-04 14:35:14,777 INFO [INIT] Starting encryption 2019-11-04 14:35:14,778 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:35:14,779 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:35:14,779 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:35:14,791 INFO [INIT] LAVA master has started. 2019-11-04 14:35:14,792 INFO [INIT] Using protocol version 3 2019-11-04 14:35:17,469 DEBUG lava-logs => PING(20) 2019-11-04 14:35:25,215 DEBUG Dhanu => PING(20) 2019-11-04 14:35:25,215 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:35:34,800 INFO scheduling health checks: 2019-11-04 14:35:34,834 INFO scheduling jobs: 2019-11-04 14:35:34,836 DEBUG - qemu 2019-11-04 14:35:34,913 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:35:34,913 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:35:34,918 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:35:36,213 INFO [INIT] Dropping privileges 2019-11-04 14:35:36,213 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:35:36,214 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:35:36,215 INFO [INIT] Marking all workers as offline 2019-11-04 14:35:36,246 INFO [INIT] Starting encryption 2019-11-04 14:35:36,247 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:35:36,247 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:35:36,248 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:35:36,262 INFO [INIT] LAVA master has started. 2019-11-04 14:35:36,262 INFO [INIT] Using protocol version 3 2019-11-04 14:35:37,478 DEBUG lava-logs => PING(20) 2019-11-04 14:35:45,233 DEBUG Dhanu => PING(20) 2019-11-04 14:35:45,234 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:35:56,266 INFO scheduling health checks: 2019-11-04 14:35:56,358 INFO scheduling jobs: 2019-11-04 14:35:56,361 DEBUG - qemu 2019-11-04 14:35:56,467 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:35:56,467 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:35:56,529 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:36:01,207 INFO [INIT] Dropping privileges 2019-11-04 14:36:01,207 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:36:01,207 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:36:01,210 INFO [INIT] Marking all workers as offline 2019-11-04 14:36:01,248 INFO [INIT] Starting encryption 2019-11-04 14:36:01,249 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:36:01,250 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:36:01,250 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:36:01,261 INFO [INIT] LAVA master has started. 2019-11-04 14:36:01,261 INFO [INIT] Using protocol version 3 2019-11-04 14:36:01,302 DEBUG lava-logs => PING(20) 2019-11-04 14:36:05,249 DEBUG Dhanu => PING(20) 2019-11-04 14:36:05,249 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:36:21,276 INFO scheduling health checks: 2019-11-04 14:36:21,305 INFO scheduling jobs: 2019-11-04 14:36:21,306 DEBUG - qemu 2019-11-04 14:36:21,385 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:36:21,385 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:36:21,390 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:36:22,739 INFO [INIT] Dropping privileges 2019-11-04 14:36:22,739 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:36:22,739 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:36:22,741 INFO [INIT] Marking all workers as offline 2019-11-04 14:36:22,772 INFO [INIT] Starting encryption 2019-11-04 14:36:22,774 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:36:22,774 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:36:22,775 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:36:22,786 INFO [INIT] LAVA master has started. 2019-11-04 14:36:22,786 INFO [INIT] Using protocol version 3 2019-11-04 14:36:25,264 DEBUG Dhanu => PING(20) 2019-11-04 14:36:25,264 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:36:41,337 DEBUG lava-logs => PING(20) 2019-11-04 14:36:42,788 INFO scheduling health checks: 2019-11-04 14:36:42,807 INFO scheduling jobs: 2019-11-04 14:36:42,809 DEBUG - qemu 2019-11-04 14:36:42,882 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:36:42,883 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:36:42,893 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:36:44,221 INFO [INIT] Dropping privileges 2019-11-04 14:36:44,222 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:36:44,222 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:36:44,224 INFO [INIT] Marking all workers as offline 2019-11-04 14:36:44,252 INFO [INIT] Starting encryption 2019-11-04 14:36:44,254 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:36:44,254 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:36:44,254 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:36:44,268 INFO [INIT] LAVA master has started. 2019-11-04 14:36:44,268 INFO [INIT] Using protocol version 3 2019-11-04 14:36:45,278 DEBUG Dhanu => PING(20) 2019-11-04 14:36:45,279 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:37:01,353 DEBUG lava-logs => PING(20) 2019-11-04 14:37:04,272 INFO scheduling health checks: 2019-11-04 14:37:04,331 INFO scheduling jobs: 2019-11-04 14:37:04,335 DEBUG - qemu 2019-11-04 14:37:04,430 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:37:04,430 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:37:04,493 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:37:10,844 INFO [INIT] Dropping privileges 2019-11-04 14:37:10,844 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:37:10,845 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:37:10,847 INFO [INIT] Marking all workers as offline 2019-11-04 14:37:10,877 INFO [INIT] Starting encryption 2019-11-04 14:37:10,879 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:37:10,879 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:37:10,880 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:37:10,895 INFO [INIT] LAVA master has started. 2019-11-04 14:37:10,895 INFO [INIT] Using protocol version 3 2019-11-04 14:37:10,973 DEBUG Dhanu => PING(20) 2019-11-04 14:37:10,974 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:37:21,366 DEBUG lava-logs => PING(20) 2019-11-04 14:37:30,904 INFO scheduling health checks: 2019-11-04 14:37:30,934 INFO scheduling jobs: 2019-11-04 14:37:30,936 DEBUG - qemu 2019-11-04 14:37:31,013 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:37:31,013 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:37:31,019 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:37:32,241 INFO [INIT] Dropping privileges 2019-11-04 14:37:32,241 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:37:32,242 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:37:32,243 INFO [INIT] Marking all workers as offline 2019-11-04 14:37:32,279 INFO [INIT] Starting encryption 2019-11-04 14:37:32,281 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:37:32,281 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:37:32,281 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:37:32,291 INFO [INIT] LAVA master has started. 2019-11-04 14:37:32,291 INFO [INIT] Using protocol version 3 2019-11-04 14:37:41,385 DEBUG lava-logs => PING(20) 2019-11-04 14:37:51,005 DEBUG Dhanu => PING(20) 2019-11-04 14:37:51,005 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:37:52,293 INFO scheduling health checks: 2019-11-04 14:37:52,314 INFO scheduling jobs: 2019-11-04 14:37:52,315 DEBUG - qemu 2019-11-04 14:37:52,393 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:37:52,393 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:37:52,398 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:37:53,744 INFO [INIT] Dropping privileges 2019-11-04 14:37:53,744 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:37:53,745 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:37:53,746 INFO [INIT] Marking all workers as offline 2019-11-04 14:37:53,781 INFO [INIT] Starting encryption 2019-11-04 14:37:53,783 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:37:53,783 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:37:53,784 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:37:53,796 INFO [INIT] LAVA master has started. 2019-11-04 14:37:53,796 INFO [INIT] Using protocol version 3 2019-11-04 14:38:01,401 DEBUG lava-logs => PING(20) 2019-11-04 14:38:11,021 DEBUG Dhanu => PING(20) 2019-11-04 14:38:11,043 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:38:13,799 INFO scheduling health checks: 2019-11-04 14:38:13,860 INFO scheduling jobs: 2019-11-04 14:38:13,863 DEBUG - qemu 2019-11-04 14:38:13,973 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:38:13,974 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:38:14,034 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:38:18,149 INFO [INIT] Dropping privileges 2019-11-04 14:38:18,149 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:38:18,149 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:38:18,151 INFO [INIT] Marking all workers as offline 2019-11-04 14:38:18,184 INFO [INIT] Starting encryption 2019-11-04 14:38:18,185 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:38:18,185 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:38:18,186 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:38:18,213 INFO [INIT] LAVA master has started. 2019-11-04 14:38:18,213 INFO [INIT] Using protocol version 3 2019-11-04 14:38:21,413 DEBUG lava-logs => PING(20) 2019-11-04 14:38:31,061 DEBUG Dhanu => PING(20) 2019-11-04 14:38:31,062 WARNING Unknown dispatcher (server crashed) 2019-11-04 14:38:38,220 INFO scheduling health checks: 2019-11-04 14:38:38,285 INFO scheduling jobs: 2019-11-04 14:38:38,286 DEBUG - qemu 2019-11-04 14:38:38,392 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-04 14:38:38,392 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-04 14:38:38,456 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-04 14:38:40,360 INFO [INIT] Dropping privileges 2019-11-04 14:38:40,360 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-04 14:38:40,360 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-04 14:38:40,362 INFO [INIT] Marking all workers as offline 2019-11-04 14:38:40,408 INFO [INIT] Starting encryption 2019-11-04 14:38:40,410 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-04 14:38:40,410 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:38:40,410 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-04 14:38:40,429 INFO [INIT] LAVA master has started. 2019-11-04 14:38:40,429 INFO [INIT] Using protocol version 3 2019-11-04 14:38:41,417 DEBUG lava-logs => PING(20) 2019-11-05 04:15:09,773 DEBUG Dhanu => PING(20) 2019-11-05 04:15:09,773 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:15:10,037 ERROR [STATE] lava-logs goes OFFLINE 2019-11-05 04:15:10,215 WARNING lava-logs is offline: can't schedule jobs 2019-11-05 04:15:15,129 DEBUG lava-logs => PING(20) 2019-11-05 04:15:29,789 DEBUG Dhanu => PING(20) 2019-11-05 04:15:30,219 INFO scheduling health checks: 2019-11-05 04:15:30,395 INFO scheduling jobs: 2019-11-05 04:15:30,397 DEBUG - qemu 2019-11-05 04:15:30,485 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:15:30,485 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:15:30,494 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:15:33,438 INFO [INIT] Dropping privileges 2019-11-05 04:15:33,438 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:15:33,438 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:15:33,441 INFO [INIT] Marking all workers as offline 2019-11-05 04:15:33,539 INFO [INIT] Starting encryption 2019-11-05 04:15:33,540 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:15:33,540 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:15:33,541 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:15:33,549 INFO [INIT] LAVA master has started. 2019-11-05 04:15:33,549 INFO [INIT] Using protocol version 3 2019-11-05 04:15:35,139 DEBUG lava-logs => PING(20) 2019-11-05 04:15:49,806 DEBUG Dhanu => PING(20) 2019-11-05 04:15:49,806 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:15:53,552 INFO scheduling health checks: 2019-11-05 04:15:53,607 INFO scheduling jobs: 2019-11-05 04:15:53,609 DEBUG - qemu 2019-11-05 04:15:53,683 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:15:53,683 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:15:53,689 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:15:56,812 INFO [INIT] Dropping privileges 2019-11-05 04:15:56,812 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:15:56,813 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:15:56,815 INFO [INIT] Marking all workers as offline 2019-11-05 04:15:56,875 INFO [INIT] Starting encryption 2019-11-05 04:15:56,877 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:15:56,877 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:15:56,877 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:15:56,887 INFO [INIT] LAVA master has started. 2019-11-05 04:15:56,888 INFO [INIT] Using protocol version 3 2019-11-05 04:15:56,910 DEBUG lava-logs => PING(20) 2019-11-05 04:16:09,821 DEBUG Dhanu => PING(20) 2019-11-05 04:16:09,856 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:16:16,892 INFO scheduling health checks: 2019-11-05 04:16:16,932 INFO scheduling jobs: 2019-11-05 04:16:16,933 DEBUG - qemu 2019-11-05 04:16:17,019 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:16:17,019 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:16:17,025 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:16:20,857 INFO [INIT] Dropping privileges 2019-11-05 04:16:20,857 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:16:20,857 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:16:20,859 INFO [INIT] Marking all workers as offline 2019-11-05 04:16:20,892 INFO [INIT] Starting encryption 2019-11-05 04:16:20,894 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:16:20,895 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:16:20,895 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:16:20,912 INFO [INIT] LAVA master has started. 2019-11-05 04:16:20,912 INFO [INIT] Using protocol version 3 2019-11-05 04:16:29,870 DEBUG Dhanu => PING(20) 2019-11-05 04:16:29,871 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:16:36,941 DEBUG lava-logs => PING(20) 2019-11-05 04:16:40,916 INFO scheduling health checks: 2019-11-05 04:16:40,958 INFO scheduling jobs: 2019-11-05 04:16:40,960 DEBUG - qemu 2019-11-05 04:16:41,033 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:16:41,033 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:16:41,038 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:16:42,224 INFO [INIT] Dropping privileges 2019-11-05 04:16:42,224 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:16:42,224 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:16:42,280 INFO [INIT] Marking all workers as offline 2019-11-05 04:16:42,404 INFO [INIT] Starting encryption 2019-11-05 04:16:42,405 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:16:42,405 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:16:42,405 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:16:42,415 INFO [INIT] LAVA master has started. 2019-11-05 04:16:42,415 INFO [INIT] Using protocol version 3 2019-11-05 04:16:49,890 DEBUG Dhanu => PING(20) 2019-11-05 04:16:49,890 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:16:56,957 DEBUG lava-logs => PING(20) 2019-11-05 04:17:02,419 INFO scheduling health checks: 2019-11-05 04:17:02,460 INFO scheduling jobs: 2019-11-05 04:17:02,462 DEBUG - qemu 2019-11-05 04:17:02,529 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:17:02,529 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:17:02,534 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:17:05,483 INFO [INIT] Dropping privileges 2019-11-05 04:17:05,483 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:17:05,483 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:17:05,485 INFO [INIT] Marking all workers as offline 2019-11-05 04:17:05,516 INFO [INIT] Starting encryption 2019-11-05 04:17:05,518 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:17:05,518 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:17:05,518 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:17:05,534 INFO [INIT] LAVA master has started. 2019-11-05 04:17:05,534 INFO [INIT] Using protocol version 3 2019-11-05 04:17:09,909 DEBUG Dhanu => PING(20) 2019-11-05 04:17:09,910 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:17:16,969 DEBUG lava-logs => PING(20) 2019-11-05 04:17:25,543 INFO scheduling health checks: 2019-11-05 04:17:25,573 INFO scheduling jobs: 2019-11-05 04:17:25,575 DEBUG - qemu 2019-11-05 04:17:25,650 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:17:25,650 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:17:25,655 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:17:28,116 INFO [INIT] Dropping privileges 2019-11-05 04:17:28,117 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:17:28,117 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:17:28,119 INFO [INIT] Marking all workers as offline 2019-11-05 04:17:28,152 INFO [INIT] Starting encryption 2019-11-05 04:17:28,153 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:17:28,153 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:17:28,153 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:17:28,169 INFO [INIT] LAVA master has started. 2019-11-05 04:17:28,170 INFO [INIT] Using protocol version 3 2019-11-05 04:17:29,929 DEBUG Dhanu => PING(20) 2019-11-05 04:17:29,930 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:17:36,990 DEBUG lava-logs => PING(20) 2019-11-05 04:17:48,181 INFO scheduling health checks: 2019-11-05 04:17:48,220 INFO scheduling jobs: 2019-11-05 04:17:48,223 DEBUG - qemu 2019-11-05 04:17:48,301 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:17:48,301 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:17:48,310 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:17:49,634 INFO [INIT] Dropping privileges 2019-11-05 04:17:49,634 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:17:49,634 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:17:49,636 INFO [INIT] Marking all workers as offline 2019-11-05 04:17:49,676 INFO [INIT] Starting encryption 2019-11-05 04:17:49,678 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:17:49,679 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:17:49,679 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:17:49,693 INFO [INIT] LAVA master has started. 2019-11-05 04:17:49,694 INFO [INIT] Using protocol version 3 2019-11-05 04:17:49,945 DEBUG Dhanu => PING(20) 2019-11-05 04:17:49,946 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:17:57,010 DEBUG lava-logs => PING(20) 2019-11-05 04:18:09,707 INFO scheduling health checks: 2019-11-05 04:18:09,744 INFO scheduling jobs: 2019-11-05 04:18:09,747 DEBUG - qemu 2019-11-05 04:18:09,821 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:18:09,821 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:18:09,826 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:18:11,127 INFO [INIT] Dropping privileges 2019-11-05 04:18:11,127 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:18:11,128 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:18:11,129 INFO [INIT] Marking all workers as offline 2019-11-05 04:18:11,201 INFO [INIT] Starting encryption 2019-11-05 04:18:11,203 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:18:11,203 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:18:11,204 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:18:11,218 INFO [INIT] LAVA master has started. 2019-11-05 04:18:11,218 INFO [INIT] Using protocol version 3 2019-11-05 04:18:11,312 DEBUG Dhanu => PING(20) 2019-11-05 04:18:11,312 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:18:17,028 DEBUG lava-logs => PING(20) 2019-11-05 04:18:31,232 INFO scheduling health checks: 2019-11-05 04:18:31,257 INFO scheduling jobs: 2019-11-05 04:18:31,258 DEBUG - qemu 2019-11-05 04:18:31,322 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:18:31,323 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:18:31,327 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:18:32,616 INFO [INIT] Dropping privileges 2019-11-05 04:18:32,616 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:18:32,616 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:18:32,618 INFO [INIT] Marking all workers as offline 2019-11-05 04:18:32,658 INFO [INIT] Starting encryption 2019-11-05 04:18:32,660 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:18:32,660 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:18:32,661 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:18:32,677 INFO [INIT] LAVA master has started. 2019-11-05 04:18:32,678 INFO [INIT] Using protocol version 3 2019-11-05 04:18:32,794 DEBUG Dhanu => PING(20) 2019-11-05 04:18:32,794 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:18:37,037 DEBUG lava-logs => PING(20) 2019-11-05 04:18:52,692 INFO scheduling health checks: 2019-11-05 04:18:52,726 INFO scheduling jobs: 2019-11-05 04:18:52,728 DEBUG - qemu 2019-11-05 04:18:52,812 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:18:52,812 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:18:52,817 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:18:55,759 INFO [INIT] Dropping privileges 2019-11-05 04:18:55,759 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:18:55,759 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:18:55,761 INFO [INIT] Marking all workers as offline 2019-11-05 04:18:55,806 INFO [INIT] Starting encryption 2019-11-05 04:18:55,807 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:18:55,808 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:18:55,809 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:18:55,825 INFO [INIT] LAVA master has started. 2019-11-05 04:18:55,825 INFO [INIT] Using protocol version 3 2019-11-05 04:18:57,052 DEBUG lava-logs => PING(20) 2019-11-05 04:19:12,828 DEBUG Dhanu => PING(20) 2019-11-05 04:19:12,829 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:19:15,828 INFO scheduling health checks: 2019-11-05 04:19:15,861 INFO scheduling jobs: 2019-11-05 04:19:15,864 DEBUG - qemu 2019-11-05 04:19:15,945 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:19:15,946 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:19:15,954 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:19:17,311 INFO [INIT] Dropping privileges 2019-11-05 04:19:17,312 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:19:17,314 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:19:17,316 INFO [INIT] Marking all workers as offline 2019-11-05 04:19:17,351 INFO [INIT] Starting encryption 2019-11-05 04:19:17,352 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:19:17,353 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:19:17,353 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:19:17,368 INFO [INIT] LAVA master has started. 2019-11-05 04:19:17,368 INFO [INIT] Using protocol version 3 2019-11-05 04:19:17,420 DEBUG lava-logs => PING(20) 2019-11-05 04:19:32,909 DEBUG Dhanu => PING(20) 2019-11-05 04:19:32,941 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:19:37,370 INFO scheduling health checks: 2019-11-05 04:19:37,442 INFO scheduling jobs: 2019-11-05 04:19:37,443 DEBUG - qemu 2019-11-05 04:19:37,613 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:19:37,613 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:19:37,704 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:19:56,116 INFO [INIT] Dropping privileges 2019-11-05 04:19:56,117 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:19:56,117 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:19:56,119 INFO [INIT] Marking all workers as offline 2019-11-05 04:19:56,163 INFO [INIT] Starting encryption 2019-11-05 04:19:56,164 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:19:56,165 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:19:56,165 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:19:56,210 INFO [INIT] LAVA master has started. 2019-11-05 04:19:56,210 INFO [INIT] Using protocol version 3 2019-11-05 04:19:56,291 DEBUG Dhanu => PING(20) 2019-11-05 04:19:56,292 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:19:57,447 DEBUG lava-logs => PING(20) 2019-11-05 04:20:16,229 INFO scheduling health checks: 2019-11-05 04:20:16,289 INFO scheduling jobs: 2019-11-05 04:20:16,293 DEBUG - qemu 2019-11-05 04:20:16,395 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:20:16,395 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:20:16,473 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:20:17,969 INFO [INIT] Dropping privileges 2019-11-05 04:20:17,969 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:20:17,970 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:20:17,971 INFO [INIT] Marking all workers as offline 2019-11-05 04:20:18,035 INFO [INIT] Starting encryption 2019-11-05 04:20:18,037 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:20:18,037 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:20:18,038 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:20:18,053 INFO [INIT] LAVA master has started. 2019-11-05 04:20:18,053 INFO [INIT] Using protocol version 3 2019-11-05 04:20:18,205 DEBUG lava-logs => PING(20) 2019-11-05 04:20:36,329 DEBUG Dhanu => PING(20) 2019-11-05 04:20:36,329 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:20:38,055 INFO scheduling health checks: 2019-11-05 04:20:38,102 INFO scheduling jobs: 2019-11-05 04:20:38,105 DEBUG - qemu 2019-11-05 04:20:38,190 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:20:38,190 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:20:38,197 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:20:39,445 INFO [INIT] Dropping privileges 2019-11-05 04:20:39,445 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:20:39,445 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:20:39,447 INFO [INIT] Marking all workers as offline 2019-11-05 04:20:39,482 INFO [INIT] Starting encryption 2019-11-05 04:20:39,483 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:20:39,484 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:20:39,485 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:20:39,502 INFO [INIT] LAVA master has started. 2019-11-05 04:20:39,502 INFO [INIT] Using protocol version 3 2019-11-05 04:20:39,566 DEBUG lava-logs => PING(20) 2019-11-05 04:20:56,346 DEBUG Dhanu => PING(20) 2019-11-05 04:20:56,347 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:20:59,506 INFO scheduling health checks: 2019-11-05 04:20:59,549 INFO scheduling jobs: 2019-11-05 04:20:59,551 DEBUG - qemu 2019-11-05 04:20:59,627 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:20:59,627 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:20:59,632 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:21:00,923 INFO [INIT] Dropping privileges 2019-11-05 04:21:00,924 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:21:00,924 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:21:00,926 INFO [INIT] Marking all workers as offline 2019-11-05 04:21:00,961 INFO [INIT] Starting encryption 2019-11-05 04:21:00,962 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:21:00,962 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:21:00,963 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:21:00,972 INFO [INIT] LAVA master has started. 2019-11-05 04:21:00,972 INFO [INIT] Using protocol version 3 2019-11-05 04:21:16,361 DEBUG Dhanu => PING(20) 2019-11-05 04:21:16,362 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:21:19,605 DEBUG lava-logs => PING(20) 2019-11-05 04:21:20,974 INFO scheduling health checks: 2019-11-05 04:21:20,995 INFO scheduling jobs: 2019-11-05 04:21:20,997 DEBUG - qemu 2019-11-05 04:21:21,090 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:21:21,091 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:21:21,099 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:21:23,045 INFO [INIT] Dropping privileges 2019-11-05 04:21:23,045 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:21:23,045 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:21:23,047 INFO [INIT] Marking all workers as offline 2019-11-05 04:21:23,107 INFO [INIT] Starting encryption 2019-11-05 04:21:23,109 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:21:23,109 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:21:23,110 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:21:23,126 INFO [INIT] LAVA master has started. 2019-11-05 04:21:23,126 INFO [INIT] Using protocol version 3 2019-11-05 04:21:36,377 DEBUG Dhanu => PING(20) 2019-11-05 04:21:36,378 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:21:39,623 DEBUG lava-logs => PING(20) 2019-11-05 04:21:43,128 INFO scheduling health checks: 2019-11-05 04:21:43,164 INFO scheduling jobs: 2019-11-05 04:21:43,166 DEBUG - qemu 2019-11-05 04:21:43,245 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:21:43,245 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:21:43,255 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:21:46,031 INFO [INIT] Dropping privileges 2019-11-05 04:21:46,031 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:21:46,031 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:21:46,033 INFO [INIT] Marking all workers as offline 2019-11-05 04:21:46,065 INFO [INIT] Starting encryption 2019-11-05 04:21:46,066 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:21:46,067 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:21:46,067 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:21:46,082 INFO [INIT] LAVA master has started. 2019-11-05 04:21:46,082 INFO [INIT] Using protocol version 3 2019-11-05 04:21:56,386 DEBUG Dhanu => PING(20) 2019-11-05 04:21:56,387 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:21:59,637 DEBUG lava-logs => PING(20) 2019-11-05 04:22:06,088 INFO scheduling health checks: 2019-11-05 04:22:06,133 INFO scheduling jobs: 2019-11-05 04:22:06,135 DEBUG - qemu 2019-11-05 04:22:06,210 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:22:06,210 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:22:06,215 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:22:07,424 INFO [INIT] Dropping privileges 2019-11-05 04:22:07,425 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:22:07,425 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:22:07,426 INFO [INIT] Marking all workers as offline 2019-11-05 04:22:07,467 INFO [INIT] Starting encryption 2019-11-05 04:22:07,469 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:22:07,469 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:22:07,470 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:22:07,488 INFO [INIT] LAVA master has started. 2019-11-05 04:22:07,489 INFO [INIT] Using protocol version 3 2019-11-05 04:22:16,408 DEBUG Dhanu => PING(20) 2019-11-05 04:22:16,409 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:22:19,653 DEBUG lava-logs => PING(20) 2019-11-05 04:22:27,496 INFO scheduling health checks: 2019-11-05 04:22:27,523 INFO scheduling jobs: 2019-11-05 04:22:27,525 DEBUG - qemu 2019-11-05 04:22:27,603 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:22:27,603 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:22:27,608 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:22:29,002 INFO [INIT] Dropping privileges 2019-11-05 04:22:29,003 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:22:29,003 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:22:29,005 INFO [INIT] Marking all workers as offline 2019-11-05 04:22:29,035 INFO [INIT] Starting encryption 2019-11-05 04:22:29,037 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:22:29,037 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:22:29,038 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:22:29,050 INFO [INIT] LAVA master has started. 2019-11-05 04:22:29,050 INFO [INIT] Using protocol version 3 2019-11-05 04:22:36,430 DEBUG Dhanu => PING(20) 2019-11-05 04:22:36,461 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:22:39,665 DEBUG lava-logs => PING(20) 2019-11-05 04:22:49,060 INFO scheduling health checks: 2019-11-05 04:22:49,103 INFO scheduling jobs: 2019-11-05 04:22:49,104 DEBUG - qemu 2019-11-05 04:22:49,182 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:22:49,182 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:22:49,186 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:22:52,663 INFO [INIT] Dropping privileges 2019-11-05 04:22:52,663 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:22:52,663 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:22:52,665 INFO [INIT] Marking all workers as offline 2019-11-05 04:22:52,693 INFO [INIT] Starting encryption 2019-11-05 04:22:52,694 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:22:52,695 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:22:52,695 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:22:52,711 INFO [INIT] LAVA master has started. 2019-11-05 04:22:52,711 INFO [INIT] Using protocol version 3 2019-11-05 04:22:56,477 DEBUG Dhanu => PING(20) 2019-11-05 04:22:56,478 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:22:59,681 DEBUG lava-logs => PING(20) 2019-11-05 04:23:12,724 INFO scheduling health checks: 2019-11-05 04:23:12,761 INFO scheduling jobs: 2019-11-05 04:23:12,763 DEBUG - qemu 2019-11-05 04:23:12,830 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:23:12,831 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:23:12,835 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:23:14,117 INFO [INIT] Dropping privileges 2019-11-05 04:23:14,117 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:23:14,117 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:23:14,119 INFO [INIT] Marking all workers as offline 2019-11-05 04:23:14,151 INFO [INIT] Starting encryption 2019-11-05 04:23:14,153 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:23:14,153 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:23:14,154 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:23:14,171 INFO [INIT] LAVA master has started. 2019-11-05 04:23:14,171 INFO [INIT] Using protocol version 3 2019-11-05 04:23:16,493 DEBUG Dhanu => PING(20) 2019-11-05 04:23:16,493 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:23:19,697 DEBUG lava-logs => PING(20) 2019-11-05 04:23:34,184 INFO scheduling health checks: 2019-11-05 04:23:34,218 INFO scheduling jobs: 2019-11-05 04:23:34,222 DEBUG - qemu 2019-11-05 04:23:34,307 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:23:34,307 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:23:34,312 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:23:35,740 INFO [INIT] Dropping privileges 2019-11-05 04:23:35,740 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:23:35,740 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:23:35,744 INFO [INIT] Marking all workers as offline 2019-11-05 04:23:35,797 INFO [INIT] Starting encryption 2019-11-05 04:23:35,801 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:23:35,802 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:23:35,802 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:23:35,820 INFO [INIT] LAVA master has started. 2019-11-05 04:23:35,820 INFO [INIT] Using protocol version 3 2019-11-05 04:23:36,510 DEBUG Dhanu => PING(20) 2019-11-05 04:23:36,511 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:23:39,713 DEBUG lava-logs => PING(20) 2019-11-05 04:23:55,833 INFO scheduling health checks: 2019-11-05 04:23:55,876 INFO scheduling jobs: 2019-11-05 04:23:55,877 DEBUG - qemu 2019-11-05 04:23:55,957 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:23:55,957 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:23:55,962 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:23:59,228 INFO [INIT] Dropping privileges 2019-11-05 04:23:59,228 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:23:59,229 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:23:59,230 INFO [INIT] Marking all workers as offline 2019-11-05 04:23:59,266 INFO [INIT] Starting encryption 2019-11-05 04:23:59,267 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:23:59,268 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:23:59,268 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:23:59,282 INFO [INIT] LAVA master has started. 2019-11-05 04:23:59,283 INFO [INIT] Using protocol version 3 2019-11-05 04:23:59,339 DEBUG Dhanu => PING(20) 2019-11-05 04:23:59,340 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:23:59,722 DEBUG lava-logs => PING(20) 2019-11-05 04:24:19,284 INFO scheduling health checks: 2019-11-05 04:24:19,356 INFO scheduling jobs: 2019-11-05 04:24:19,359 DEBUG - qemu 2019-11-05 04:24:19,433 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:24:19,433 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:24:19,438 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:24:20,694 INFO [INIT] Dropping privileges 2019-11-05 04:24:20,695 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:24:20,695 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:24:20,697 INFO [INIT] Marking all workers as offline 2019-11-05 04:24:20,757 INFO [INIT] Starting encryption 2019-11-05 04:24:20,759 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:24:20,759 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:24:20,760 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:24:20,776 INFO [INIT] LAVA master has started. 2019-11-05 04:24:20,776 INFO [INIT] Using protocol version 3 2019-11-05 04:24:20,858 DEBUG lava-logs => PING(20) 2019-11-05 04:24:39,374 DEBUG Dhanu => PING(20) 2019-11-05 04:24:39,374 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:24:40,777 INFO scheduling health checks: 2019-11-05 04:24:40,802 INFO scheduling jobs: 2019-11-05 04:24:40,804 DEBUG - qemu 2019-11-05 04:24:40,868 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:24:40,869 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:24:40,876 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:24:42,170 INFO [INIT] Dropping privileges 2019-11-05 04:24:42,170 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:24:42,170 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:24:42,172 INFO [INIT] Marking all workers as offline 2019-11-05 04:24:42,203 INFO [INIT] Starting encryption 2019-11-05 04:24:42,204 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:24:42,205 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:24:42,205 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:24:42,222 INFO [INIT] LAVA master has started. 2019-11-05 04:24:42,222 INFO [INIT] Using protocol version 3 2019-11-05 04:24:59,389 DEBUG Dhanu => PING(20) 2019-11-05 04:24:59,390 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:25:00,893 DEBUG lava-logs => PING(20) 2019-11-05 04:25:02,224 INFO scheduling health checks: 2019-11-05 04:25:02,260 INFO scheduling jobs: 2019-11-05 04:25:02,261 DEBUG - qemu 2019-11-05 04:25:02,341 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:25:02,341 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:25:02,348 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:25:06,545 INFO [INIT] Dropping privileges 2019-11-05 04:25:06,546 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:25:06,546 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:25:06,549 INFO [INIT] Marking all workers as offline 2019-11-05 04:25:06,583 INFO [INIT] Starting encryption 2019-11-05 04:25:06,585 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:25:06,586 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:25:06,586 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:25:06,600 INFO [INIT] LAVA master has started. 2019-11-05 04:25:06,601 INFO [INIT] Using protocol version 3 2019-11-05 04:25:19,403 DEBUG Dhanu => PING(20) 2019-11-05 04:25:19,403 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:25:20,900 DEBUG lava-logs => PING(20) 2019-11-05 04:25:26,604 INFO scheduling health checks: 2019-11-05 04:25:26,640 INFO scheduling jobs: 2019-11-05 04:25:26,643 DEBUG - qemu 2019-11-05 04:25:26,718 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:25:26,718 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:25:26,723 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:25:29,320 INFO [INIT] Dropping privileges 2019-11-05 04:25:29,320 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:25:29,320 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:25:29,322 INFO [INIT] Marking all workers as offline 2019-11-05 04:25:29,363 INFO [INIT] Starting encryption 2019-11-05 04:25:29,365 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:25:29,366 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:25:29,366 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:25:29,382 INFO [INIT] LAVA master has started. 2019-11-05 04:25:29,382 INFO [INIT] Using protocol version 3 2019-11-05 04:25:39,424 DEBUG Dhanu => PING(20) 2019-11-05 04:25:39,425 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:25:40,913 DEBUG lava-logs => PING(20) 2019-11-05 04:25:49,388 INFO scheduling health checks: 2019-11-05 04:25:49,431 INFO scheduling jobs: 2019-11-05 04:25:49,434 DEBUG - qemu 2019-11-05 04:25:49,514 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:25:49,514 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:25:49,521 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:25:52,217 INFO [INIT] Dropping privileges 2019-11-05 04:25:52,218 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:25:52,218 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:25:52,219 INFO [INIT] Marking all workers as offline 2019-11-05 04:25:52,254 INFO [INIT] Starting encryption 2019-11-05 04:25:52,256 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:25:52,256 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:25:52,256 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:25:52,274 INFO [INIT] LAVA master has started. 2019-11-05 04:25:52,274 INFO [INIT] Using protocol version 3 2019-11-05 04:25:59,440 DEBUG Dhanu => PING(20) 2019-11-05 04:25:59,441 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:26:00,933 DEBUG lava-logs => PING(20) 2019-11-05 04:26:12,284 INFO scheduling health checks: 2019-11-05 04:26:12,311 INFO scheduling jobs: 2019-11-05 04:26:12,313 DEBUG - qemu 2019-11-05 04:26:12,389 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:26:12,389 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:26:12,394 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:26:13,647 INFO [INIT] Dropping privileges 2019-11-05 04:26:13,647 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:26:13,648 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:26:13,651 INFO [INIT] Marking all workers as offline 2019-11-05 04:26:13,678 INFO [INIT] Starting encryption 2019-11-05 04:26:13,680 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:26:13,680 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:26:13,680 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:26:13,695 INFO [INIT] LAVA master has started. 2019-11-05 04:26:13,696 INFO [INIT] Using protocol version 3 2019-11-05 04:26:19,454 DEBUG Dhanu => PING(20) 2019-11-05 04:26:19,455 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:26:20,951 DEBUG lava-logs => PING(20) 2019-11-05 04:26:33,704 INFO scheduling health checks: 2019-11-05 04:26:33,746 INFO scheduling jobs: 2019-11-05 04:26:33,749 DEBUG - qemu 2019-11-05 04:26:33,826 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:26:33,826 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:26:33,831 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:26:35,332 INFO [INIT] Dropping privileges 2019-11-05 04:26:35,332 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:26:35,332 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:26:35,334 INFO [INIT] Marking all workers as offline 2019-11-05 04:26:35,392 INFO [INIT] Starting encryption 2019-11-05 04:26:35,394 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:26:35,395 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:26:35,395 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:26:35,413 INFO [INIT] LAVA master has started. 2019-11-05 04:26:35,413 INFO [INIT] Using protocol version 3 2019-11-05 04:26:39,473 DEBUG Dhanu => PING(20) 2019-11-05 04:26:39,474 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:26:40,968 DEBUG lava-logs => PING(20) 2019-11-05 04:26:55,415 INFO scheduling health checks: 2019-11-05 04:26:55,448 INFO scheduling jobs: 2019-11-05 04:26:55,450 DEBUG - qemu 2019-11-05 04:26:55,523 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:26:55,523 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:26:55,528 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:26:56,932 INFO [INIT] Dropping privileges 2019-11-05 04:26:56,935 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:26:56,935 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:26:56,937 INFO [INIT] Marking all workers as offline 2019-11-05 04:26:56,971 INFO [INIT] Starting encryption 2019-11-05 04:26:56,972 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:26:56,975 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:26:56,975 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:26:56,991 INFO [INIT] LAVA master has started. 2019-11-05 04:26:56,991 INFO [INIT] Using protocol version 3 2019-11-05 04:26:59,492 DEBUG Dhanu => PING(20) 2019-11-05 04:26:59,493 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:27:00,982 DEBUG lava-logs => PING(20) 2019-11-05 04:27:17,004 INFO scheduling health checks: 2019-11-05 04:27:17,028 INFO scheduling jobs: 2019-11-05 04:27:17,030 DEBUG - qemu 2019-11-05 04:27:17,106 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:27:17,106 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:27:17,111 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:27:18,501 INFO [INIT] Dropping privileges 2019-11-05 04:27:18,501 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:27:18,502 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:27:18,503 INFO [INIT] Marking all workers as offline 2019-11-05 04:27:18,540 INFO [INIT] Starting encryption 2019-11-05 04:27:18,542 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:27:18,543 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:27:18,543 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:27:18,559 INFO [INIT] LAVA master has started. 2019-11-05 04:27:18,559 INFO [INIT] Using protocol version 3 2019-11-05 04:27:19,508 DEBUG Dhanu => PING(20) 2019-11-05 04:27:19,509 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:27:20,996 DEBUG lava-logs => PING(20) 2019-11-05 04:27:38,559 INFO scheduling health checks: 2019-11-05 04:27:38,597 INFO scheduling jobs: 2019-11-05 04:27:38,598 DEBUG - qemu 2019-11-05 04:27:38,667 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:27:38,667 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:27:38,672 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:27:39,978 INFO [INIT] Dropping privileges 2019-11-05 04:27:39,978 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:27:39,978 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:27:39,980 INFO [INIT] Marking all workers as offline 2019-11-05 04:27:40,020 INFO [INIT] Starting encryption 2019-11-05 04:27:40,022 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:27:40,022 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:27:40,023 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:27:40,038 INFO [INIT] LAVA master has started. 2019-11-05 04:27:40,038 INFO [INIT] Using protocol version 3 2019-11-05 04:27:40,232 DEBUG Dhanu => PING(20) 2019-11-05 04:27:40,232 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:27:41,011 DEBUG lava-logs => PING(20) 2019-11-05 04:28:00,056 INFO scheduling health checks: 2019-11-05 04:28:00,088 INFO scheduling jobs: 2019-11-05 04:28:00,089 DEBUG - qemu 2019-11-05 04:28:00,163 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:28:00,163 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:28:00,169 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:28:01,560 INFO [INIT] Dropping privileges 2019-11-05 04:28:01,560 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:28:01,560 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:28:01,562 INFO [INIT] Marking all workers as offline 2019-11-05 04:28:01,600 INFO [INIT] Starting encryption 2019-11-05 04:28:01,601 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:28:01,601 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:28:01,602 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:28:01,621 INFO [INIT] LAVA master has started. 2019-11-05 04:28:01,621 INFO [INIT] Using protocol version 3 2019-11-05 04:28:01,687 DEBUG Dhanu => PING(20) 2019-11-05 04:28:01,687 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:28:01,721 DEBUG lava-logs => PING(20) 2019-11-05 04:28:21,642 INFO scheduling health checks: 2019-11-05 04:28:21,679 INFO scheduling jobs: 2019-11-05 04:28:21,682 DEBUG - qemu 2019-11-05 04:28:21,755 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:28:21,756 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:28:21,760 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:28:22,872 INFO [INIT] Dropping privileges 2019-11-05 04:28:22,872 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:28:22,872 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:28:22,874 INFO [INIT] Marking all workers as offline 2019-11-05 04:28:22,913 INFO [INIT] Starting encryption 2019-11-05 04:28:22,914 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:28:22,915 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:28:22,915 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:28:22,931 INFO [INIT] LAVA master has started. 2019-11-05 04:28:22,931 INFO [INIT] Using protocol version 3 2019-11-05 04:28:41,727 DEBUG Dhanu => PING(20) 2019-11-05 04:28:41,727 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:28:41,780 DEBUG lava-logs => PING(20) 2019-11-05 04:28:42,933 INFO scheduling health checks: 2019-11-05 04:28:42,958 INFO scheduling jobs: 2019-11-05 04:28:42,960 DEBUG - qemu 2019-11-05 04:28:43,026 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:28:43,026 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:28:43,031 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:28:44,290 INFO [INIT] Dropping privileges 2019-11-05 04:28:44,290 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:28:44,291 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:28:44,292 INFO [INIT] Marking all workers as offline 2019-11-05 04:28:44,326 INFO [INIT] Starting encryption 2019-11-05 04:28:44,329 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:28:44,330 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:28:44,330 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:28:44,345 INFO [INIT] LAVA master has started. 2019-11-05 04:28:44,346 INFO [INIT] Using protocol version 3 2019-11-05 04:29:01,749 DEBUG Dhanu => PING(20) 2019-11-05 04:29:01,749 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:29:01,805 DEBUG lava-logs => PING(20) 2019-11-05 04:29:04,348 INFO scheduling health checks: 2019-11-05 04:29:04,372 INFO scheduling jobs: 2019-11-05 04:29:04,373 DEBUG - qemu 2019-11-05 04:29:04,446 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:29:04,446 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:29:04,450 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:29:05,631 INFO [INIT] Dropping privileges 2019-11-05 04:29:05,632 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:29:05,632 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:29:05,633 INFO [INIT] Marking all workers as offline 2019-11-05 04:29:05,673 INFO [INIT] Starting encryption 2019-11-05 04:29:05,674 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:29:05,675 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:29:05,675 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:29:05,691 INFO [INIT] LAVA master has started. 2019-11-05 04:29:05,692 INFO [INIT] Using protocol version 3 2019-11-05 04:29:21,765 DEBUG Dhanu => PING(20) 2019-11-05 04:29:21,766 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:29:21,840 DEBUG lava-logs => PING(20) 2019-11-05 04:29:25,696 INFO scheduling health checks: 2019-11-05 04:29:25,729 INFO scheduling jobs: 2019-11-05 04:29:25,731 DEBUG - qemu 2019-11-05 04:29:25,809 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:29:25,809 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} [Previous line repeated 971 more times] File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 1, in top-level template code {% extends 'qemu01.jinja2' %} File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:29:25,814 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:29:27,126 INFO [INIT] Dropping privileges 2019-11-05 04:29:27,126 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:29:27,127 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:29:27,128 INFO [INIT] Marking all workers as offline 2019-11-05 04:29:27,152 INFO [INIT] Starting encryption 2019-11-05 04:29:27,152 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:29:27,153 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:29:27,153 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:29:27,163 INFO [INIT] LAVA master has started. 2019-11-05 04:29:27,163 INFO [INIT] Using protocol version 3 2019-11-05 04:29:41,786 DEBUG Dhanu => PING(20) 2019-11-05 04:29:41,786 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:29:41,856 DEBUG lava-logs => PING(20) 2019-11-05 04:29:47,168 INFO scheduling health checks: 2019-11-05 04:29:47,198 INFO scheduling jobs: 2019-11-05 04:29:47,200 DEBUG - qemu 2019-11-05 04:29:47,268 ERROR [CLOSE] Unknown exception raised, leaving! 2019-11-05 04:29:47,268 ERROR maximum recursion depth exceeded in comparison Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 687, in handle self.main_loop(options) File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 777, in main_loop schedule(self.logger) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 40, in schedule jobs.extend(schedule_jobs(logger, available_devices)) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 194, in schedule_jobs schedule_jobs_for_device_type(logger, dt, available_devices[dt.name]) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 222, in schedule_jobs_for_device_type new_job = schedule_jobs_for_device(logger, device) File "/usr/lib/python3/dist-packages/lava_scheduler_app/scheduler.py", line 245, in schedule_jobs_for_device if not device.is_valid(): File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 669, in is_valid rendered = self.load_configuration() File "/usr/lib/python3/dist-packages/lava_scheduler_app/models.py", line 822, in load_configuration device_template = template.render(**job_ctx) File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/etc/lava-server/dispatcher-config/devices/qemu01.jinja2", line 3, in top-level template code {% set memory = 1024 %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} File "/etc/lava-server/dispatcher-config/device-types/qemu.jinja2", line 1, in top-level template code {% extends 'qemu.jinja2' %} [Previous line repeated 971 more times] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 351, in get return self[key] File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 400, in __getitem__ rv = self._mapping[key] RecursionError: maximum recursion depth exceeded in comparison 2019-11-05 04:29:47,276 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:29:48,397 INFO [INIT] Dropping privileges 2019-11-05 04:29:48,397 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:29:48,397 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:29:48,399 INFO [INIT] Marking all workers as offline 2019-11-05 04:29:48,432 INFO [INIT] Starting encryption 2019-11-05 04:29:48,434 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:29:48,434 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:29:48,435 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:29:48,447 INFO [INIT] LAVA master has started. 2019-11-05 04:29:48,447 INFO [INIT] Using protocol version 3 2019-11-05 04:30:01,804 DEBUG Dhanu => PING(20) 2019-11-05 04:30:01,804 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:30:01,877 DEBUG lava-logs => PING(20) 2019-11-05 04:30:08,454 INFO scheduling health checks: 2019-11-05 04:30:08,489 INFO scheduling jobs: 2019-11-05 04:30:08,491 DEBUG - qemu 2019-11-05 04:30:08,522 DEBUG Good ??? Bad (Invalid device configuration for qemu01) 2019-11-05 04:30:21,825 DEBUG Dhanu => PING(20) 2019-11-05 04:30:21,897 DEBUG lava-logs => PING(20) 2019-11-05 04:30:22,567 INFO [POLL] Received a signal, leaving 2019-11-05 04:30:22,567 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:30:23,599 INFO [INIT] Dropping privileges 2019-11-05 04:30:23,599 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:30:23,599 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:30:23,601 INFO [INIT] Marking all workers as offline 2019-11-05 04:30:23,635 INFO [INIT] Starting encryption 2019-11-05 04:30:23,637 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:30:23,637 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:30:23,638 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:30:23,653 INFO [INIT] LAVA master has started. 2019-11-05 04:30:23,654 INFO [INIT] Using protocol version 3 2019-11-05 04:30:41,496 DEBUG lava-logs => PING(20) 2019-11-05 04:30:41,514 DEBUG lava-logs => PING(20) 2019-11-05 04:30:41,846 DEBUG Dhanu => PING(20) 2019-11-05 04:30:41,846 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:30:43,656 INFO scheduling health checks: 2019-11-05 04:30:43,671 INFO scheduling jobs: 2019-11-05 04:31:01,533 DEBUG lava-logs => PING(20) 2019-11-05 04:31:01,867 DEBUG Dhanu => PING(20) 2019-11-05 04:31:03,679 INFO scheduling health checks: 2019-11-05 04:31:03,688 INFO scheduling jobs: 2019-11-05 04:31:21,553 DEBUG lava-logs => PING(20) 2019-11-05 04:31:21,886 DEBUG Dhanu => PING(20) 2019-11-05 04:31:23,699 INFO scheduling health checks: 2019-11-05 04:31:23,710 INFO scheduling jobs: 2019-11-05 04:31:41,573 DEBUG lava-logs => PING(20) 2019-11-05 04:31:41,905 DEBUG Dhanu => PING(20) 2019-11-05 04:31:43,727 INFO scheduling health checks: 2019-11-05 04:31:43,732 INFO scheduling jobs: 2019-11-05 04:32:01,593 DEBUG lava-logs => PING(20) 2019-11-05 04:32:01,923 DEBUG Dhanu => PING(20) 2019-11-05 04:32:03,743 INFO scheduling health checks: 2019-11-05 04:32:03,753 INFO scheduling jobs: 2019-11-05 04:32:21,616 DEBUG lava-logs => PING(20) 2019-11-05 04:32:21,941 DEBUG Dhanu => PING(20) 2019-11-05 04:32:23,767 INFO scheduling health checks: 2019-11-05 04:32:23,775 INFO scheduling jobs: 2019-11-05 04:32:41,630 DEBUG lava-logs => PING(20) 2019-11-05 04:32:41,963 DEBUG Dhanu => PING(20) 2019-11-05 04:32:43,787 INFO scheduling health checks: 2019-11-05 04:32:43,798 INFO scheduling jobs: 2019-11-05 04:33:01,653 DEBUG lava-logs => PING(20) 2019-11-05 04:33:01,984 DEBUG Dhanu => PING(20) 2019-11-05 04:33:03,810 INFO scheduling health checks: 2019-11-05 04:33:03,820 INFO scheduling jobs: 2019-11-05 04:33:21,668 DEBUG lava-logs => PING(20) 2019-11-05 04:33:22,003 DEBUG Dhanu => PING(20) 2019-11-05 04:33:23,833 INFO scheduling health checks: 2019-11-05 04:33:23,845 INFO scheduling jobs: 2019-11-05 04:33:41,681 DEBUG lava-logs => PING(20) 2019-11-05 04:33:42,025 DEBUG Dhanu => PING(20) 2019-11-05 04:33:43,856 INFO scheduling health checks: 2019-11-05 04:33:43,867 INFO scheduling jobs: 2019-11-05 04:34:01,696 DEBUG lava-logs => PING(20) 2019-11-05 04:34:02,041 DEBUG Dhanu => PING(20) 2019-11-05 04:34:03,878 INFO scheduling health checks: 2019-11-05 04:34:03,887 INFO scheduling jobs: 2019-11-05 04:34:21,716 DEBUG lava-logs => PING(20) 2019-11-05 04:34:22,063 DEBUG Dhanu => PING(20) 2019-11-05 04:34:23,898 INFO scheduling health checks: 2019-11-05 04:34:23,905 INFO scheduling jobs: 2019-11-05 04:34:41,737 DEBUG lava-logs => PING(20) 2019-11-05 04:34:42,081 DEBUG Dhanu => PING(20) 2019-11-05 04:34:43,920 INFO scheduling health checks: 2019-11-05 04:34:43,930 INFO scheduling jobs: 2019-11-05 04:35:01,760 DEBUG lava-logs => PING(20) 2019-11-05 04:35:02,097 DEBUG Dhanu => PING(20) 2019-11-05 04:35:03,940 INFO scheduling health checks: 2019-11-05 04:35:03,949 INFO scheduling jobs: 2019-11-05 04:35:21,781 DEBUG lava-logs => PING(20) 2019-11-05 04:35:22,117 DEBUG Dhanu => PING(20) 2019-11-05 04:35:23,964 INFO scheduling health checks: 2019-11-05 04:35:23,975 INFO scheduling jobs: 2019-11-05 04:35:41,837 DEBUG lava-logs => PING(20) 2019-11-05 04:35:42,132 DEBUG Dhanu => PING(20) 2019-11-05 04:35:43,990 INFO scheduling health checks: 2019-11-05 04:35:44,000 INFO scheduling jobs: 2019-11-05 04:36:01,860 DEBUG lava-logs => PING(20) 2019-11-05 04:36:02,150 DEBUG Dhanu => PING(20) 2019-11-05 04:36:04,014 INFO scheduling health checks: 2019-11-05 04:36:04,026 INFO scheduling jobs: 2019-11-05 04:36:21,878 DEBUG lava-logs => PING(20) 2019-11-05 04:36:22,171 DEBUG Dhanu => PING(20) 2019-11-05 04:36:24,038 INFO scheduling health checks: 2019-11-05 04:36:24,051 INFO scheduling jobs: 2019-11-05 04:36:41,899 DEBUG lava-logs => PING(20) 2019-11-05 04:36:42,190 DEBUG Dhanu => PING(20) 2019-11-05 04:36:44,062 INFO scheduling health checks: 2019-11-05 04:36:44,069 INFO scheduling jobs: 2019-11-05 04:37:01,919 DEBUG lava-logs => PING(20) 2019-11-05 04:37:02,209 DEBUG Dhanu => PING(20) 2019-11-05 04:37:04,081 INFO scheduling health checks: 2019-11-05 04:37:04,088 INFO scheduling jobs: 2019-11-05 04:37:21,941 DEBUG lava-logs => PING(20) 2019-11-05 04:37:22,227 DEBUG Dhanu => PING(20) 2019-11-05 04:37:24,107 INFO scheduling health checks: 2019-11-05 04:37:24,116 INFO scheduling jobs: 2019-11-05 04:37:42,037 DEBUG lava-logs => PING(20) 2019-11-05 04:37:42,245 DEBUG Dhanu => PING(20) 2019-11-05 04:37:44,128 INFO scheduling health checks: 2019-11-05 04:37:44,138 INFO scheduling jobs: 2019-11-05 04:38:02,057 DEBUG lava-logs => PING(20) 2019-11-05 04:38:02,261 DEBUG Dhanu => PING(20) 2019-11-05 04:38:04,151 INFO scheduling health checks: 2019-11-05 04:38:04,161 INFO scheduling jobs: 2019-11-05 04:38:22,079 DEBUG lava-logs => PING(20) 2019-11-05 04:38:22,283 DEBUG Dhanu => PING(20) 2019-11-05 04:38:24,170 INFO scheduling health checks: 2019-11-05 04:38:24,181 INFO scheduling jobs: 2019-11-05 04:38:42,097 DEBUG lava-logs => PING(20) 2019-11-05 04:38:42,303 DEBUG Dhanu => PING(20) 2019-11-05 04:38:44,193 INFO scheduling health checks: 2019-11-05 04:38:44,204 INFO scheduling jobs: 2019-11-05 04:39:02,117 DEBUG lava-logs => PING(20) 2019-11-05 04:39:02,325 DEBUG Dhanu => PING(20) 2019-11-05 04:39:04,215 INFO scheduling health checks: 2019-11-05 04:39:04,224 INFO scheduling jobs: 2019-11-05 04:39:22,141 DEBUG lava-logs => PING(20) 2019-11-05 04:39:22,348 DEBUG Dhanu => PING(20) 2019-11-05 04:39:24,236 INFO scheduling health checks: 2019-11-05 04:39:24,247 INFO scheduling jobs: 2019-11-05 04:39:42,165 DEBUG lava-logs => PING(20) 2019-11-05 04:39:42,367 DEBUG Dhanu => PING(20) 2019-11-05 04:39:44,257 INFO scheduling health checks: 2019-11-05 04:39:44,263 INFO scheduling jobs: 2019-11-05 04:40:02,183 DEBUG lava-logs => PING(20) 2019-11-05 04:40:02,392 DEBUG Dhanu => PING(20) 2019-11-05 04:40:04,273 INFO scheduling health checks: 2019-11-05 04:40:04,284 INFO scheduling jobs: 2019-11-05 04:40:22,201 DEBUG lava-logs => PING(20) 2019-11-05 04:40:22,416 DEBUG Dhanu => PING(20) 2019-11-05 04:40:24,299 INFO scheduling health checks: 2019-11-05 04:40:24,309 INFO scheduling jobs: 2019-11-05 04:40:42,224 DEBUG lava-logs => PING(20) 2019-11-05 04:40:42,437 DEBUG Dhanu => PING(20) 2019-11-05 04:40:44,316 INFO scheduling health checks: 2019-11-05 04:40:44,328 INFO scheduling jobs: 2019-11-05 04:41:02,245 DEBUG lava-logs => PING(20) 2019-11-05 04:41:02,460 DEBUG Dhanu => PING(20) 2019-11-05 04:41:04,339 INFO scheduling health checks: 2019-11-05 04:41:04,350 INFO scheduling jobs: 2019-11-05 04:41:22,266 DEBUG lava-logs => PING(20) 2019-11-05 04:41:22,481 DEBUG Dhanu => PING(20) 2019-11-05 04:41:24,361 INFO scheduling health checks: 2019-11-05 04:41:24,371 INFO scheduling jobs: 2019-11-05 04:41:42,285 DEBUG lava-logs => PING(20) 2019-11-05 04:41:42,501 DEBUG Dhanu => PING(20) 2019-11-05 04:41:44,383 INFO scheduling health checks: 2019-11-05 04:41:44,392 INFO scheduling jobs: 2019-11-05 04:42:02,307 DEBUG lava-logs => PING(20) 2019-11-05 04:42:02,521 DEBUG Dhanu => PING(20) 2019-11-05 04:42:04,409 INFO scheduling health checks: 2019-11-05 04:42:04,419 INFO scheduling jobs: 2019-11-05 04:42:22,330 DEBUG lava-logs => PING(20) 2019-11-05 04:42:22,541 DEBUG Dhanu => PING(20) 2019-11-05 04:42:24,433 INFO scheduling health checks: 2019-11-05 04:42:24,444 INFO scheduling jobs: 2019-11-05 04:42:42,346 DEBUG lava-logs => PING(20) 2019-11-05 04:42:42,562 DEBUG Dhanu => PING(20) 2019-11-05 04:42:44,460 INFO scheduling health checks: 2019-11-05 04:42:44,471 INFO scheduling jobs: 2019-11-05 04:43:02,366 DEBUG lava-logs => PING(20) 2019-11-05 04:43:02,583 DEBUG Dhanu => PING(20) 2019-11-05 04:43:04,483 INFO scheduling health checks: 2019-11-05 04:43:04,494 INFO scheduling jobs: 2019-11-05 04:43:22,388 DEBUG lava-logs => PING(20) 2019-11-05 04:43:22,605 DEBUG Dhanu => PING(20) 2019-11-05 04:43:24,507 INFO scheduling health checks: 2019-11-05 04:43:24,518 INFO scheduling jobs: 2019-11-05 04:43:42,409 DEBUG lava-logs => PING(20) 2019-11-05 04:43:42,625 DEBUG Dhanu => PING(20) 2019-11-05 04:43:44,528 INFO scheduling health checks: 2019-11-05 04:43:44,539 INFO scheduling jobs: 2019-11-05 04:44:02,432 DEBUG lava-logs => PING(20) 2019-11-05 04:44:02,647 DEBUG Dhanu => PING(20) 2019-11-05 04:44:04,549 INFO scheduling health checks: 2019-11-05 04:44:04,558 INFO scheduling jobs: 2019-11-05 04:44:22,452 DEBUG lava-logs => PING(20) 2019-11-05 04:44:22,665 DEBUG Dhanu => PING(20) 2019-11-05 04:44:24,570 INFO scheduling health checks: 2019-11-05 04:44:24,581 INFO scheduling jobs: 2019-11-05 04:44:42,473 DEBUG lava-logs => PING(20) 2019-11-05 04:44:42,681 DEBUG Dhanu => PING(20) 2019-11-05 04:44:44,591 INFO scheduling health checks: 2019-11-05 04:44:44,602 INFO scheduling jobs: 2019-11-05 04:45:02,493 DEBUG lava-logs => PING(20) 2019-11-05 04:45:02,701 DEBUG Dhanu => PING(20) 2019-11-05 04:45:04,615 INFO scheduling health checks: 2019-11-05 04:45:04,626 INFO scheduling jobs: 2019-11-05 04:45:22,516 DEBUG lava-logs => PING(20) 2019-11-05 04:45:22,725 DEBUG Dhanu => PING(20) 2019-11-05 04:45:24,640 INFO scheduling health checks: 2019-11-05 04:45:24,651 INFO scheduling jobs: 2019-11-05 04:45:42,536 DEBUG lava-logs => PING(20) 2019-11-05 04:45:42,743 DEBUG Dhanu => PING(20) 2019-11-05 04:45:44,663 INFO scheduling health checks: 2019-11-05 04:45:44,673 INFO scheduling jobs: 2019-11-05 04:46:01,743 DEBUG lava-logs => PING(20) 2019-11-05 04:46:01,791 DEBUG lava-logs => PING(20) 2019-11-05 04:46:02,765 DEBUG Dhanu => PING(20) 2019-11-05 04:46:04,686 INFO scheduling health checks: 2019-11-05 04:46:04,692 INFO scheduling jobs: 2019-11-05 04:46:21,809 DEBUG lava-logs => PING(20) 2019-11-05 04:46:22,781 DEBUG Dhanu => PING(20) 2019-11-05 04:46:24,703 INFO scheduling health checks: 2019-11-05 04:46:24,712 INFO scheduling jobs: 2019-11-05 04:46:41,829 DEBUG lava-logs => PING(20) 2019-11-05 04:46:42,803 DEBUG Dhanu => PING(20) 2019-11-05 04:46:44,725 INFO scheduling health checks: 2019-11-05 04:46:44,736 INFO scheduling jobs: 2019-11-05 04:47:01,849 DEBUG lava-logs => PING(20) 2019-11-05 04:47:02,823 DEBUG Dhanu => PING(20) 2019-11-05 04:47:04,748 INFO scheduling health checks: 2019-11-05 04:47:04,761 INFO scheduling jobs: 2019-11-05 04:47:21,869 DEBUG lava-logs => PING(20) 2019-11-05 04:47:22,841 DEBUG Dhanu => PING(20) 2019-11-05 04:47:24,777 INFO scheduling health checks: 2019-11-05 04:47:24,788 INFO scheduling jobs: 2019-11-05 04:47:41,889 DEBUG lava-logs => PING(20) 2019-11-05 04:47:42,860 DEBUG Dhanu => PING(20) 2019-11-05 04:47:44,802 INFO scheduling health checks: 2019-11-05 04:47:44,810 INFO scheduling jobs: 2019-11-05 04:48:01,910 DEBUG lava-logs => PING(20) 2019-11-05 04:48:02,879 DEBUG Dhanu => PING(20) 2019-11-05 04:48:04,824 INFO scheduling health checks: 2019-11-05 04:48:04,835 INFO scheduling jobs: 2019-11-05 04:48:21,931 DEBUG lava-logs => PING(20) 2019-11-05 04:48:22,901 DEBUG Dhanu => PING(20) 2019-11-05 04:48:24,847 INFO scheduling health checks: 2019-11-05 04:48:24,854 INFO scheduling jobs: 2019-11-05 04:48:41,951 DEBUG lava-logs => PING(20) 2019-11-05 04:48:42,917 DEBUG Dhanu => PING(20) 2019-11-05 04:48:44,868 INFO scheduling health checks: 2019-11-05 04:48:44,880 INFO scheduling jobs: 2019-11-05 04:49:01,973 DEBUG lava-logs => PING(20) 2019-11-05 04:49:02,937 DEBUG Dhanu => PING(20) 2019-11-05 04:49:04,892 INFO scheduling health checks: 2019-11-05 04:49:04,901 INFO scheduling jobs: 2019-11-05 04:49:21,988 DEBUG lava-logs => PING(20) 2019-11-05 04:49:22,954 DEBUG Dhanu => PING(20) 2019-11-05 04:49:24,913 INFO scheduling health checks: 2019-11-05 04:49:24,924 INFO scheduling jobs: 2019-11-05 04:49:42,008 DEBUG lava-logs => PING(20) 2019-11-05 04:49:42,976 DEBUG Dhanu => PING(20) 2019-11-05 04:49:44,938 INFO scheduling health checks: 2019-11-05 04:49:44,948 INFO scheduling jobs: 2019-11-05 04:50:02,030 DEBUG lava-logs => PING(20) 2019-11-05 04:50:02,995 DEBUG Dhanu => PING(20) 2019-11-05 04:50:04,958 INFO scheduling health checks: 2019-11-05 04:50:04,967 INFO scheduling jobs: 2019-11-05 04:50:22,054 DEBUG lava-logs => PING(20) 2019-11-05 04:50:23,017 DEBUG Dhanu => PING(20) 2019-11-05 04:50:24,976 INFO scheduling health checks: 2019-11-05 04:50:24,982 INFO scheduling jobs: 2019-11-05 04:50:42,074 DEBUG lava-logs => PING(20) 2019-11-05 04:50:43,037 DEBUG Dhanu => PING(20) 2019-11-05 04:50:44,993 INFO scheduling health checks: 2019-11-05 04:50:45,003 INFO scheduling jobs: 2019-11-05 04:51:02,090 DEBUG lava-logs => PING(20) 2019-11-05 04:51:03,057 DEBUG Dhanu => PING(20) 2019-11-05 04:51:05,014 INFO scheduling health checks: 2019-11-05 04:51:05,024 INFO scheduling jobs: 2019-11-05 04:51:22,111 DEBUG lava-logs => PING(20) 2019-11-05 04:51:23,077 DEBUG Dhanu => PING(20) 2019-11-05 04:51:25,040 INFO scheduling health checks: 2019-11-05 04:51:25,050 INFO scheduling jobs: 2019-11-05 04:51:42,129 DEBUG lava-logs => PING(20) 2019-11-05 04:51:43,097 DEBUG Dhanu => PING(20) 2019-11-05 04:51:45,060 INFO scheduling health checks: 2019-11-05 04:51:45,064 INFO scheduling jobs: 2019-11-05 04:52:02,152 DEBUG lava-logs => PING(20) 2019-11-05 04:52:03,113 DEBUG Dhanu => PING(20) 2019-11-05 04:52:05,071 INFO scheduling health checks: 2019-11-05 04:52:05,082 INFO scheduling jobs: 2019-11-05 04:52:22,170 DEBUG lava-logs => PING(20) 2019-11-05 04:52:23,139 DEBUG Dhanu => PING(20) 2019-11-05 04:52:25,094 INFO scheduling health checks: 2019-11-05 04:52:25,103 INFO scheduling jobs: 2019-11-05 04:52:42,192 DEBUG lava-logs => PING(20) 2019-11-05 04:52:43,161 DEBUG Dhanu => PING(20) 2019-11-05 04:52:45,115 INFO scheduling health checks: 2019-11-05 04:52:45,126 INFO scheduling jobs: 2019-11-05 04:53:02,212 DEBUG lava-logs => PING(20) 2019-11-05 04:53:03,180 DEBUG Dhanu => PING(20) 2019-11-05 04:53:05,138 INFO scheduling health checks: 2019-11-05 04:53:05,149 INFO scheduling jobs: 2019-11-05 04:53:22,235 DEBUG lava-logs => PING(20) 2019-11-05 04:53:23,201 DEBUG Dhanu => PING(20) 2019-11-05 04:53:25,160 INFO scheduling health checks: 2019-11-05 04:53:25,170 INFO scheduling jobs: 2019-11-05 04:53:42,256 DEBUG lava-logs => PING(20) 2019-11-05 04:53:43,224 DEBUG Dhanu => PING(20) 2019-11-05 04:53:45,180 INFO scheduling health checks: 2019-11-05 04:53:45,191 INFO scheduling jobs: 2019-11-05 04:54:02,280 DEBUG lava-logs => PING(20) 2019-11-05 04:54:03,249 DEBUG Dhanu => PING(20) 2019-11-05 04:54:05,204 INFO scheduling health checks: 2019-11-05 04:54:05,215 INFO scheduling jobs: 2019-11-05 04:54:22,297 DEBUG lava-logs => PING(20) 2019-11-05 04:54:23,265 DEBUG Dhanu => PING(20) 2019-11-05 04:54:25,227 INFO scheduling health checks: 2019-11-05 04:54:25,238 INFO scheduling jobs: 2019-11-05 04:54:42,321 DEBUG lava-logs => PING(20) 2019-11-05 04:54:43,286 DEBUG Dhanu => PING(20) 2019-11-05 04:54:45,247 INFO scheduling health checks: 2019-11-05 04:54:45,254 INFO scheduling jobs: 2019-11-05 04:55:02,337 DEBUG lava-logs => PING(20) 2019-11-05 04:55:03,305 DEBUG Dhanu => PING(20) 2019-11-05 04:55:05,266 INFO scheduling health checks: 2019-11-05 04:55:05,279 INFO scheduling jobs: 2019-11-05 04:55:22,353 DEBUG lava-logs => PING(20) 2019-11-05 04:55:23,321 DEBUG Dhanu => PING(20) 2019-11-05 04:55:25,288 INFO scheduling health checks: 2019-11-05 04:55:25,298 INFO scheduling jobs: 2019-11-05 04:55:42,373 DEBUG lava-logs => PING(20) 2019-11-05 04:55:43,339 DEBUG Dhanu => PING(20) 2019-11-05 04:55:45,312 INFO scheduling health checks: 2019-11-05 04:55:45,321 INFO scheduling jobs: 2019-11-05 04:56:02,393 DEBUG lava-logs => PING(20) 2019-11-05 04:56:03,363 DEBUG Dhanu => PING(20) 2019-11-05 04:56:05,336 INFO scheduling health checks: 2019-11-05 04:56:05,347 INFO scheduling jobs: 2019-11-05 04:56:22,413 DEBUG lava-logs => PING(20) 2019-11-05 04:56:23,388 DEBUG Dhanu => PING(20) 2019-11-05 04:56:25,358 INFO scheduling health checks: 2019-11-05 04:56:25,369 INFO scheduling jobs: 2019-11-05 04:56:42,436 DEBUG lava-logs => PING(20) 2019-11-05 04:56:43,413 DEBUG Dhanu => PING(20) 2019-11-05 04:56:45,380 INFO scheduling health checks: 2019-11-05 04:56:45,391 INFO scheduling jobs: 2019-11-05 04:57:03,436 DEBUG Dhanu => PING(20) 2019-11-05 04:57:05,404 INFO scheduling health checks: 2019-11-05 04:57:05,411 INFO scheduling jobs: 2019-11-05 04:57:09,835 DEBUG lava-logs => PING(20) 2019-11-05 04:57:09,856 DEBUG lava-logs => PING(20) 2019-11-05 04:57:15,482 INFO [POLL] Received a signal, leaving 2019-11-05 04:57:15,482 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:57:16,914 INFO [INIT] Dropping privileges 2019-11-05 04:57:16,914 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:57:16,914 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:57:16,916 INFO [INIT] Marking all workers as offline 2019-11-05 04:57:16,991 INFO [INIT] Starting encryption 2019-11-05 04:57:16,993 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:57:16,993 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:57:16,994 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:57:17,011 INFO [INIT] LAVA master has started. 2019-11-05 04:57:17,011 INFO [INIT] Using protocol version 3 2019-11-05 04:57:23,453 DEBUG Dhanu => PING(20) 2019-11-05 04:57:23,454 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:57:29,866 DEBUG lava-logs => PING(20) 2019-11-05 04:57:30,843 INFO Dhanu => HELLO 2019-11-05 04:57:30,843 WARNING Dispatcher has RESTARTED 2019-11-05 04:57:35,849 DEBUG Dhanu => PING(20) 2019-11-05 04:57:37,013 INFO scheduling health checks: 2019-11-05 04:57:37,028 INFO scheduling jobs: 2019-11-05 04:57:49,885 DEBUG lava-logs => PING(20) 2019-11-05 04:57:55,865 DEBUG Dhanu => PING(20) 2019-11-05 04:57:57,038 INFO scheduling health checks: 2019-11-05 04:57:57,049 INFO scheduling jobs: 2019-11-05 04:58:09,897 DEBUG lava-logs => PING(20) 2019-11-05 04:58:15,886 DEBUG Dhanu => PING(20) 2019-11-05 04:58:17,060 INFO scheduling health checks: 2019-11-05 04:58:17,071 INFO scheduling jobs: 2019-11-05 04:58:29,916 DEBUG lava-logs => PING(20) 2019-11-05 04:58:34,808 INFO [POLL] Received a signal, leaving 2019-11-05 04:58:34,808 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 04:58:35,798 INFO [INIT] Dropping privileges 2019-11-05 04:58:35,798 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 04:58:35,798 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 04:58:35,800 INFO [INIT] Marking all workers as offline 2019-11-05 04:58:35,843 INFO [INIT] Starting encryption 2019-11-05 04:58:35,845 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 04:58:35,845 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:58:35,846 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 04:58:35,863 INFO [INIT] LAVA master has started. 2019-11-05 04:58:35,863 INFO [INIT] Using protocol version 3 2019-11-05 04:58:35,902 DEBUG Dhanu => PING(20) 2019-11-05 04:58:35,902 WARNING Unknown dispatcher (server crashed) 2019-11-05 04:58:36,683 INFO Dhanu => HELLO 2019-11-05 04:58:36,683 WARNING Dispatcher has RESTARTED 2019-11-05 04:58:41,689 DEBUG Dhanu => PING(20) 2019-11-05 04:58:55,868 WARNING lava-logs is offline: can't schedule jobs 2019-11-05 04:59:00,651 DEBUG lava-logs => PING(20) 2019-11-05 04:59:00,689 DEBUG lava-logs => PING(20) 2019-11-05 04:59:01,705 DEBUG Dhanu => PING(20) 2019-11-05 04:59:15,880 INFO scheduling health checks: 2019-11-05 04:59:15,887 INFO scheduling jobs: 2019-11-05 04:59:20,708 DEBUG lava-logs => PING(20) 2019-11-05 04:59:21,721 DEBUG Dhanu => PING(20) 2019-11-05 04:59:35,904 INFO scheduling health checks: 2019-11-05 04:59:35,909 INFO scheduling jobs: 2019-11-05 04:59:40,726 DEBUG lava-logs => PING(20) 2019-11-05 04:59:41,741 DEBUG Dhanu => PING(20) 2019-11-05 04:59:55,924 INFO scheduling health checks: 2019-11-05 04:59:55,930 INFO scheduling jobs: 2019-11-05 05:00:00,747 DEBUG lava-logs => PING(20) 2019-11-05 05:00:01,761 DEBUG Dhanu => PING(20) 2019-11-05 05:00:15,945 INFO scheduling health checks: 2019-11-05 05:00:15,956 INFO scheduling jobs: 2019-11-05 05:00:20,768 DEBUG lava-logs => PING(20) 2019-11-05 05:00:21,784 DEBUG Dhanu => PING(20) 2019-11-05 05:00:35,967 INFO scheduling health checks: 2019-11-05 05:00:35,978 INFO scheduling jobs: 2019-11-05 05:00:40,789 DEBUG lava-logs => PING(20) 2019-11-05 05:00:41,803 DEBUG Dhanu => PING(20) 2019-11-05 05:00:55,990 INFO scheduling health checks: 2019-11-05 05:00:56,002 INFO scheduling jobs: 2019-11-05 05:01:00,810 DEBUG lava-logs => PING(20) 2019-11-05 05:01:01,825 DEBUG Dhanu => PING(20) 2019-11-05 05:01:16,013 INFO scheduling health checks: 2019-11-05 05:01:16,019 INFO scheduling jobs: 2019-11-05 05:01:20,829 DEBUG lava-logs => PING(20) 2019-11-05 05:01:21,845 DEBUG Dhanu => PING(20) 2019-11-05 05:01:36,030 INFO scheduling health checks: 2019-11-05 05:01:36,041 INFO scheduling jobs: 2019-11-05 05:01:40,851 DEBUG lava-logs => PING(20) 2019-11-05 05:01:41,863 DEBUG Dhanu => PING(20) 2019-11-05 05:01:56,054 INFO scheduling health checks: 2019-11-05 05:01:56,066 INFO scheduling jobs: 2019-11-05 05:02:00,869 DEBUG lava-logs => PING(20) 2019-11-05 05:02:01,883 DEBUG Dhanu => PING(20) 2019-11-05 05:02:16,076 INFO scheduling health checks: 2019-11-05 05:02:16,087 INFO scheduling jobs: 2019-11-05 05:02:20,889 DEBUG lava-logs => PING(20) 2019-11-05 05:02:21,901 DEBUG Dhanu => PING(20) 2019-11-05 05:02:36,104 INFO scheduling health checks: 2019-11-05 05:02:36,144 INFO scheduling jobs: 2019-11-05 05:02:36,146 DEBUG - qemu 2019-11-05 05:02:40,909 DEBUG lava-logs => PING(20) 2019-11-05 05:02:41,917 DEBUG Dhanu => PING(20) 2019-11-05 05:02:56,180 INFO scheduling health checks: 2019-11-05 05:02:56,201 INFO scheduling jobs: 2019-11-05 05:02:56,203 DEBUG - qemu 2019-11-05 05:03:00,930 DEBUG lava-logs => PING(20) 2019-11-05 05:03:01,933 DEBUG Dhanu => PING(20) 2019-11-05 05:03:16,236 INFO scheduling health checks: 2019-11-05 05:03:16,258 INFO scheduling jobs: 2019-11-05 05:03:16,261 DEBUG - qemu 2019-11-05 05:03:20,949 DEBUG lava-logs => PING(20) 2019-11-05 05:03:21,951 DEBUG Dhanu => PING(20) 2019-11-05 05:03:36,294 INFO scheduling health checks: 2019-11-05 05:03:36,316 INFO scheduling jobs: 2019-11-05 05:03:36,319 DEBUG - qemu 2019-11-05 05:03:40,969 DEBUG lava-logs => PING(20) 2019-11-05 05:03:41,971 DEBUG Dhanu => PING(20) 2019-11-05 05:03:56,348 INFO scheduling health checks: 2019-11-05 05:03:56,374 INFO scheduling jobs: 2019-11-05 05:03:56,377 DEBUG - qemu 2019-11-05 05:04:00,989 DEBUG lava-logs => PING(20) 2019-11-05 05:04:01,992 DEBUG Dhanu => PING(20) 2019-11-05 05:04:16,412 INFO scheduling health checks: 2019-11-05 05:04:16,433 INFO scheduling jobs: 2019-11-05 05:04:16,434 DEBUG - qemu 2019-11-05 05:04:21,009 DEBUG lava-logs => PING(20) 2019-11-05 05:04:22,010 DEBUG Dhanu => PING(20) 2019-11-05 05:04:36,463 INFO scheduling health checks: 2019-11-05 05:04:36,586 INFO scheduling jobs: 2019-11-05 05:04:36,589 DEBUG - qemu 2019-11-05 05:04:41,029 DEBUG lava-logs => PING(20) 2019-11-05 05:04:42,027 DEBUG Dhanu => PING(20) 2019-11-05 05:04:56,634 INFO scheduling health checks: 2019-11-05 05:04:56,669 INFO scheduling jobs: 2019-11-05 05:04:56,672 DEBUG - qemu 2019-11-05 05:05:01,052 DEBUG lava-logs => PING(20) 2019-11-05 05:05:02,045 DEBUG Dhanu => PING(20) 2019-11-05 05:05:16,704 INFO scheduling health checks: 2019-11-05 05:05:16,737 INFO scheduling jobs: 2019-11-05 05:05:16,739 DEBUG - qemu 2019-11-05 05:05:21,069 DEBUG lava-logs => PING(20) 2019-11-05 05:05:22,068 DEBUG Dhanu => PING(20) 2019-11-05 05:05:36,886 INFO scheduling health checks: 2019-11-05 05:05:36,906 INFO scheduling jobs: 2019-11-05 05:05:36,907 DEBUG - qemu 2019-11-05 05:05:41,203 DEBUG lava-logs => PING(20) 2019-11-05 05:05:42,089 DEBUG Dhanu => PING(20) 2019-11-05 05:05:56,925 INFO scheduling health checks: 2019-11-05 05:05:56,985 INFO scheduling jobs: 2019-11-05 05:05:56,989 DEBUG - qemu 2019-11-05 05:06:01,225 DEBUG lava-logs => PING(20) 2019-11-05 05:06:02,111 DEBUG Dhanu => PING(20) 2019-11-05 05:06:17,021 INFO scheduling health checks: 2019-11-05 05:06:17,065 INFO scheduling jobs: 2019-11-05 05:06:17,067 DEBUG - qemu 2019-11-05 05:06:21,320 DEBUG lava-logs => PING(20) 2019-11-05 05:06:22,129 DEBUG Dhanu => PING(20) 2019-11-05 05:06:37,099 INFO scheduling health checks: 2019-11-05 05:06:37,134 INFO scheduling jobs: 2019-11-05 05:06:37,137 DEBUG - qemu 2019-11-05 05:06:41,341 DEBUG lava-logs => PING(20) 2019-11-05 05:06:42,146 DEBUG Dhanu => PING(20) 2019-11-05 05:06:57,168 INFO scheduling health checks: 2019-11-05 05:06:57,214 INFO scheduling jobs: 2019-11-05 05:06:57,216 DEBUG - qemu 2019-11-05 05:07:01,365 DEBUG lava-logs => PING(20) 2019-11-05 05:07:02,165 DEBUG Dhanu => PING(20) 2019-11-05 05:07:17,248 INFO scheduling health checks: 2019-11-05 05:07:17,282 INFO scheduling jobs: 2019-11-05 05:07:17,284 DEBUG - qemu 2019-11-05 05:07:21,381 DEBUG lava-logs => PING(20) 2019-11-05 05:07:22,185 DEBUG Dhanu => PING(20) 2019-11-05 05:07:37,318 INFO scheduling health checks: 2019-11-05 05:07:37,351 INFO scheduling jobs: 2019-11-05 05:07:37,355 DEBUG - qemu 2019-11-05 05:07:41,401 DEBUG lava-logs => PING(20) 2019-11-05 05:07:42,205 DEBUG Dhanu => PING(20) 2019-11-05 05:07:57,385 INFO scheduling health checks: 2019-11-05 05:07:57,408 INFO scheduling jobs: 2019-11-05 05:07:57,411 DEBUG - qemu 2019-11-05 05:08:01,421 DEBUG lava-logs => PING(20) 2019-11-05 05:08:02,225 DEBUG Dhanu => PING(20) 2019-11-05 05:08:17,444 INFO scheduling health checks: 2019-11-05 05:08:17,466 INFO scheduling jobs: 2019-11-05 05:08:17,468 DEBUG - qemu 2019-11-05 05:08:21,445 DEBUG lava-logs => PING(20) 2019-11-05 05:08:22,245 DEBUG Dhanu => PING(20) 2019-11-05 05:08:37,506 INFO scheduling health checks: 2019-11-05 05:08:37,523 INFO scheduling jobs: 2019-11-05 05:08:37,528 DEBUG - qemu 2019-11-05 05:08:41,465 DEBUG lava-logs => PING(20) 2019-11-05 05:08:42,264 DEBUG Dhanu => PING(20) 2019-11-05 05:08:57,558 INFO scheduling health checks: 2019-11-05 05:08:57,592 INFO scheduling jobs: 2019-11-05 05:08:57,594 DEBUG - qemu 2019-11-05 05:09:01,485 DEBUG lava-logs => PING(20) 2019-11-05 05:09:02,284 DEBUG Dhanu => PING(20) 2019-11-05 05:09:17,628 INFO scheduling health checks: 2019-11-05 05:09:17,650 INFO scheduling jobs: 2019-11-05 05:09:17,652 DEBUG - qemu 2019-11-05 05:09:21,507 DEBUG lava-logs => PING(20) 2019-11-05 05:09:22,309 DEBUG Dhanu => PING(20) 2019-11-05 05:09:37,683 INFO scheduling health checks: 2019-11-05 05:09:37,718 INFO scheduling jobs: 2019-11-05 05:09:37,722 DEBUG - qemu 2019-11-05 05:09:41,528 DEBUG lava-logs => PING(20) 2019-11-05 05:09:42,330 DEBUG Dhanu => PING(20) 2019-11-05 05:09:57,752 INFO scheduling health checks: 2019-11-05 05:09:57,787 INFO scheduling jobs: 2019-11-05 05:09:57,790 DEBUG - qemu 2019-11-05 05:10:01,542 DEBUG lava-logs => PING(20) 2019-11-05 05:10:02,352 DEBUG Dhanu => PING(20) 2019-11-05 05:10:17,820 INFO scheduling health checks: 2019-11-05 05:10:17,866 INFO scheduling jobs: 2019-11-05 05:10:17,871 DEBUG - qemu 2019-11-05 05:10:21,565 DEBUG lava-logs => PING(20) 2019-11-05 05:10:22,373 DEBUG Dhanu => PING(20) 2019-11-05 05:10:30,468 INFO [POLL] Received a signal, leaving 2019-11-05 05:10:30,468 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 05:10:37,834 INFO [INIT] Dropping privileges 2019-11-05 05:10:37,835 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 05:10:37,835 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 05:10:37,837 INFO [INIT] Marking all workers as offline 2019-11-05 05:10:37,881 INFO [INIT] Starting encryption 2019-11-05 05:10:37,882 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 05:10:37,883 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:10:37,883 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:10:37,899 INFO [INIT] LAVA master has started. 2019-11-05 05:10:37,899 INFO [INIT] Using protocol version 3 2019-11-05 05:10:37,925 INFO Dhanu => HELLO 2019-11-05 05:10:37,926 WARNING New dispatcher 2019-11-05 05:10:41,581 DEBUG lava-logs => PING(20) 2019-11-05 05:10:42,933 DEBUG Dhanu => PING(20) 2019-11-05 05:10:57,912 INFO scheduling health checks: 2019-11-05 05:10:57,950 INFO scheduling jobs: 2019-11-05 05:10:57,951 DEBUG - qemu 2019-11-05 05:11:01,601 DEBUG lava-logs => PING(20) 2019-11-05 05:11:02,955 DEBUG Dhanu => PING(20) 2019-11-05 05:11:17,980 INFO scheduling health checks: 2019-11-05 05:11:18,006 INFO scheduling jobs: 2019-11-05 05:11:18,008 DEBUG - qemu 2019-11-05 05:11:21,621 DEBUG lava-logs => PING(20) 2019-11-05 05:11:22,975 DEBUG Dhanu => PING(20) 2019-11-05 05:11:38,024 INFO scheduling health checks: 2019-11-05 05:11:38,075 INFO scheduling jobs: 2019-11-05 05:11:38,076 DEBUG - qemu 2019-11-05 05:11:41,641 DEBUG lava-logs => PING(20) 2019-11-05 05:11:42,997 DEBUG Dhanu => PING(20) 2019-11-05 05:11:58,095 INFO scheduling health checks: 2019-11-05 05:11:58,132 INFO scheduling jobs: 2019-11-05 05:11:58,135 DEBUG - qemu 2019-11-05 05:12:01,664 DEBUG lava-logs => PING(20) 2019-11-05 05:12:03,021 DEBUG Dhanu => PING(20) 2019-11-05 05:12:18,165 INFO scheduling health checks: 2019-11-05 05:12:18,189 INFO scheduling jobs: 2019-11-05 05:12:18,191 DEBUG - qemu 2019-11-05 05:12:21,685 DEBUG lava-logs => PING(20) 2019-11-05 05:12:23,046 DEBUG Dhanu => PING(20) 2019-11-05 05:12:38,207 INFO scheduling health checks: 2019-11-05 05:12:38,236 INFO scheduling jobs: 2019-11-05 05:12:38,238 DEBUG - qemu 2019-11-05 05:12:41,708 DEBUG lava-logs => PING(20) 2019-11-05 05:12:43,068 DEBUG Dhanu => PING(20) 2019-11-05 05:12:58,267 INFO scheduling health checks: 2019-11-05 05:12:58,293 INFO scheduling jobs: 2019-11-05 05:12:58,296 DEBUG - qemu 2019-11-05 05:13:01,729 DEBUG lava-logs => PING(20) 2019-11-05 05:13:03,088 DEBUG Dhanu => PING(20) 2019-11-05 05:13:18,329 INFO scheduling health checks: 2019-11-05 05:13:18,362 INFO scheduling jobs: 2019-11-05 05:13:18,363 DEBUG - qemu 2019-11-05 05:13:21,749 DEBUG lava-logs => PING(20) 2019-11-05 05:13:23,106 DEBUG Dhanu => PING(20) 2019-11-05 05:13:38,386 INFO scheduling health checks: 2019-11-05 05:13:38,408 INFO scheduling jobs: 2019-11-05 05:13:38,411 DEBUG - qemu 2019-11-05 05:13:41,770 DEBUG lava-logs => PING(20) 2019-11-05 05:13:43,129 DEBUG Dhanu => PING(20) 2019-11-05 05:13:58,444 INFO scheduling health checks: 2019-11-05 05:13:58,466 INFO scheduling jobs: 2019-11-05 05:13:58,469 DEBUG - qemu 2019-11-05 05:14:01,788 DEBUG lava-logs => PING(20) 2019-11-05 05:14:03,146 DEBUG Dhanu => PING(20) 2019-11-05 05:14:18,503 INFO scheduling health checks: 2019-11-05 05:14:18,535 INFO scheduling jobs: 2019-11-05 05:14:18,537 DEBUG - qemu 2019-11-05 05:14:21,835 DEBUG lava-logs => PING(20) 2019-11-05 05:14:23,170 DEBUG Dhanu => PING(20) 2019-11-05 05:14:35,149 INFO Dhanu => HELLO 2019-11-05 05:14:35,149 WARNING Dispatcher has RESTARTED 2019-11-05 05:14:38,430 INFO [POLL] Received a signal, leaving 2019-11-05 05:14:38,430 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 05:14:39,748 INFO [INIT] Dropping privileges 2019-11-05 05:14:39,748 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 05:14:39,749 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 05:14:39,750 INFO [INIT] Marking all workers as offline 2019-11-05 05:14:39,782 INFO [INIT] Starting encryption 2019-11-05 05:14:39,784 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 05:14:39,785 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:14:39,785 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:14:39,799 INFO [INIT] LAVA master has started. 2019-11-05 05:14:39,799 INFO [INIT] Using protocol version 3 2019-11-05 05:14:40,153 DEBUG Dhanu => PING(20) 2019-11-05 05:14:40,153 WARNING Unknown dispatcher (server crashed) 2019-11-05 05:14:41,849 DEBUG lava-logs => PING(20) 2019-11-05 05:14:57,247 DEBUG lava-logs => PING(20) 2019-11-05 05:14:57,271 DEBUG lava-logs => PING(20) 2019-11-05 05:14:59,800 INFO scheduling health checks: 2019-11-05 05:14:59,818 DEBUG - qemu 2019-11-05 05:14:59,818 DEBUG -> qemu01 (Idle, Good) 2019-11-05 05:14:59,838 DEBUG Good ??? Bad (Invalid device configuration for qemu01) 2019-11-05 05:14:59,850 INFO scheduling jobs: 2019-11-05 05:15:00,169 DEBUG Dhanu => PING(20) 2019-11-05 05:15:17,289 DEBUG lava-logs => PING(20) 2019-11-05 05:15:19,863 INFO scheduling health checks: 2019-11-05 05:15:19,874 INFO scheduling jobs: 2019-11-05 05:15:20,185 DEBUG Dhanu => PING(20) 2019-11-05 05:15:37,307 DEBUG lava-logs => PING(20) 2019-11-05 05:15:39,885 INFO scheduling health checks: 2019-11-05 05:15:39,896 INFO scheduling jobs: 2019-11-05 05:15:40,205 DEBUG Dhanu => PING(20) 2019-11-05 05:15:57,331 DEBUG lava-logs => PING(20) 2019-11-05 05:15:59,911 INFO scheduling health checks: 2019-11-05 05:15:59,922 INFO scheduling jobs: 2019-11-05 05:16:00,229 DEBUG Dhanu => PING(20) 2019-11-05 05:16:17,354 DEBUG lava-logs => PING(20) 2019-11-05 05:16:19,933 INFO scheduling health checks: 2019-11-05 05:16:19,944 INFO scheduling jobs: 2019-11-05 05:16:20,254 DEBUG Dhanu => PING(20) 2019-11-05 05:16:37,456 DEBUG lava-logs => PING(20) 2019-11-05 05:16:39,961 INFO scheduling health checks: 2019-11-05 05:16:39,974 INFO scheduling jobs: 2019-11-05 05:16:40,275 DEBUG Dhanu => PING(20) 2019-11-05 05:16:57,506 DEBUG lava-logs => PING(20) 2019-11-05 05:16:59,987 INFO scheduling health checks: 2019-11-05 05:16:59,998 INFO scheduling jobs: 2019-11-05 05:17:00,298 DEBUG Dhanu => PING(20) 2019-11-05 05:17:17,529 DEBUG lava-logs => PING(20) 2019-11-05 05:17:20,010 INFO scheduling health checks: 2019-11-05 05:17:20,019 INFO scheduling jobs: 2019-11-05 05:17:20,321 DEBUG Dhanu => PING(20) 2019-11-05 05:17:37,552 DEBUG lava-logs => PING(20) 2019-11-05 05:17:40,029 INFO scheduling health checks: 2019-11-05 05:17:40,033 INFO scheduling jobs: 2019-11-05 05:17:40,337 DEBUG Dhanu => PING(20) 2019-11-05 05:17:57,569 DEBUG lava-logs => PING(20) 2019-11-05 05:18:00,041 INFO scheduling health checks: 2019-11-05 05:18:00,046 INFO scheduling jobs: 2019-11-05 05:18:00,387 DEBUG Dhanu => PING(20) 2019-11-05 05:18:17,591 DEBUG lava-logs => PING(20) 2019-11-05 05:18:20,061 INFO scheduling health checks: 2019-11-05 05:18:20,072 INFO scheduling jobs: 2019-11-05 05:18:20,405 DEBUG Dhanu => PING(20) 2019-11-05 05:18:37,611 DEBUG lava-logs => PING(20) 2019-11-05 05:18:40,084 INFO scheduling health checks: 2019-11-05 05:18:40,095 INFO scheduling jobs: 2019-11-05 05:18:40,425 DEBUG Dhanu => PING(20) 2019-11-05 05:18:57,622 DEBUG lava-logs => PING(20) 2019-11-05 05:19:00,109 INFO scheduling health checks: 2019-11-05 05:19:00,165 INFO scheduling jobs: 2019-11-05 05:19:00,448 DEBUG Dhanu => PING(20) 2019-11-05 05:19:20,183 INFO scheduling health checks: 2019-11-05 05:19:20,187 INFO scheduling jobs: 2019-11-05 05:19:20,521 DEBUG Dhanu => PING(20) 2019-11-05 05:19:26,465 DEBUG lava-logs => PING(20) 2019-11-05 05:19:26,487 DEBUG lava-logs => PING(20) 2019-11-05 05:19:40,203 INFO scheduling health checks: 2019-11-05 05:19:40,206 INFO scheduling jobs: 2019-11-05 05:19:40,540 DEBUG Dhanu => PING(20) 2019-11-05 05:19:46,510 DEBUG lava-logs => PING(20) 2019-11-05 05:20:00,222 INFO scheduling health checks: 2019-11-05 05:20:00,228 INFO scheduling jobs: 2019-11-05 05:20:00,562 DEBUG Dhanu => PING(20) 2019-11-05 05:20:06,532 DEBUG lava-logs => PING(20) 2019-11-05 05:20:20,242 INFO scheduling health checks: 2019-11-05 05:20:20,246 INFO scheduling jobs: 2019-11-05 05:20:20,592 DEBUG Dhanu => PING(20) 2019-11-05 05:20:26,576 DEBUG lava-logs => PING(20) 2019-11-05 05:20:40,262 INFO scheduling health checks: 2019-11-05 05:20:40,266 INFO scheduling jobs: 2019-11-05 05:20:40,614 DEBUG Dhanu => PING(20) 2019-11-05 05:20:46,593 DEBUG lava-logs => PING(20) 2019-11-05 05:21:00,283 INFO scheduling health checks: 2019-11-05 05:21:00,286 INFO scheduling jobs: 2019-11-05 05:21:00,669 DEBUG Dhanu => PING(20) 2019-11-05 05:21:06,613 DEBUG lava-logs => PING(20) 2019-11-05 05:21:20,297 INFO scheduling health checks: 2019-11-05 05:21:20,301 INFO scheduling jobs: 2019-11-05 05:21:20,689 DEBUG Dhanu => PING(20) 2019-11-05 05:21:26,635 DEBUG lava-logs => PING(20) 2019-11-05 05:21:40,316 INFO scheduling health checks: 2019-11-05 05:21:40,321 INFO scheduling jobs: 2019-11-05 05:21:40,710 DEBUG Dhanu => PING(20) 2019-11-05 05:21:46,658 DEBUG lava-logs => PING(20) 2019-11-05 05:22:00,338 INFO scheduling health checks: 2019-11-05 05:22:00,343 INFO scheduling jobs: 2019-11-05 05:22:00,733 DEBUG Dhanu => PING(20) 2019-11-05 05:22:06,681 DEBUG lava-logs => PING(20) 2019-11-05 05:22:20,358 INFO scheduling health checks: 2019-11-05 05:22:20,362 INFO scheduling jobs: 2019-11-05 05:22:20,758 DEBUG Dhanu => PING(20) 2019-11-05 05:22:26,697 DEBUG lava-logs => PING(20) 2019-11-05 05:22:40,376 INFO scheduling health checks: 2019-11-05 05:22:40,380 INFO scheduling jobs: 2019-11-05 05:22:40,782 DEBUG Dhanu => PING(20) 2019-11-05 05:22:46,756 DEBUG lava-logs => PING(20) 2019-11-05 05:23:00,397 INFO scheduling health checks: 2019-11-05 05:23:00,401 INFO scheduling jobs: 2019-11-05 05:23:00,929 DEBUG Dhanu => PING(20) 2019-11-05 05:23:06,777 DEBUG lava-logs => PING(20) 2019-11-05 05:23:20,414 INFO scheduling health checks: 2019-11-05 05:23:20,418 INFO scheduling jobs: 2019-11-05 05:23:20,948 DEBUG Dhanu => PING(20) 2019-11-05 05:23:26,797 DEBUG lava-logs => PING(20) 2019-11-05 05:23:40,432 INFO scheduling health checks: 2019-11-05 05:23:40,436 INFO scheduling jobs: 2019-11-05 05:23:40,965 DEBUG Dhanu => PING(20) 2019-11-05 05:23:46,820 DEBUG lava-logs => PING(20) 2019-11-05 05:24:00,450 INFO scheduling health checks: 2019-11-05 05:24:00,453 INFO scheduling jobs: 2019-11-05 05:24:00,987 DEBUG Dhanu => PING(20) 2019-11-05 05:24:06,837 DEBUG lava-logs => PING(20) 2019-11-05 05:24:20,468 INFO scheduling health checks: 2019-11-05 05:24:20,473 INFO scheduling jobs: 2019-11-05 05:24:21,005 DEBUG Dhanu => PING(20) 2019-11-05 05:24:26,858 DEBUG lava-logs => PING(20) 2019-11-05 05:24:40,490 INFO scheduling health checks: 2019-11-05 05:24:40,494 INFO scheduling jobs: 2019-11-05 05:24:41,021 DEBUG Dhanu => PING(20) 2019-11-05 05:24:46,872 DEBUG lava-logs => PING(20) 2019-11-05 05:25:00,512 INFO scheduling health checks: 2019-11-05 05:25:00,516 INFO scheduling jobs: 2019-11-05 05:25:01,037 DEBUG Dhanu => PING(20) 2019-11-05 05:25:06,893 DEBUG lava-logs => PING(20) 2019-11-05 05:25:20,534 INFO scheduling health checks: 2019-11-05 05:25:20,540 INFO scheduling jobs: 2019-11-05 05:25:21,056 DEBUG Dhanu => PING(20) 2019-11-05 05:25:26,912 DEBUG lava-logs => PING(20) 2019-11-05 05:25:40,557 INFO scheduling health checks: 2019-11-05 05:25:40,562 INFO scheduling jobs: 2019-11-05 05:25:41,081 DEBUG Dhanu => PING(20) 2019-11-05 05:25:46,928 DEBUG lava-logs => PING(20) 2019-11-05 05:26:00,579 INFO scheduling health checks: 2019-11-05 05:26:00,585 INFO scheduling jobs: 2019-11-05 05:26:01,104 DEBUG Dhanu => PING(20) 2019-11-05 05:26:06,947 DEBUG lava-logs => PING(20) 2019-11-05 05:26:20,601 INFO scheduling health checks: 2019-11-05 05:26:20,605 INFO scheduling jobs: 2019-11-05 05:26:21,125 DEBUG Dhanu => PING(20) 2019-11-05 05:26:26,971 DEBUG lava-logs => PING(20) 2019-11-05 05:26:40,620 INFO scheduling health checks: 2019-11-05 05:26:40,625 INFO scheduling jobs: 2019-11-05 05:26:41,145 DEBUG Dhanu => PING(20) 2019-11-05 05:26:46,994 DEBUG lava-logs => PING(20) 2019-11-05 05:27:00,640 INFO scheduling health checks: 2019-11-05 05:27:00,644 INFO scheduling jobs: 2019-11-05 05:27:01,169 DEBUG Dhanu => PING(20) 2019-11-05 05:27:07,016 DEBUG lava-logs => PING(20) 2019-11-05 05:27:20,659 INFO scheduling health checks: 2019-11-05 05:27:20,665 INFO scheduling jobs: 2019-11-05 05:27:21,185 DEBUG Dhanu => PING(20) 2019-11-05 05:27:27,037 DEBUG lava-logs => PING(20) 2019-11-05 05:27:40,681 INFO scheduling health checks: 2019-11-05 05:27:40,685 INFO scheduling jobs: 2019-11-05 05:27:41,209 DEBUG Dhanu => PING(20) 2019-11-05 05:27:47,057 DEBUG lava-logs => PING(20) 2019-11-05 05:28:00,704 INFO scheduling health checks: 2019-11-05 05:28:00,707 INFO scheduling jobs: 2019-11-05 05:28:01,227 DEBUG Dhanu => PING(20) 2019-11-05 05:28:07,093 DEBUG lava-logs => PING(20) 2019-11-05 05:28:20,723 INFO scheduling health checks: 2019-11-05 05:28:20,727 INFO scheduling jobs: 2019-11-05 05:28:21,249 DEBUG Dhanu => PING(20) 2019-11-05 05:28:27,117 DEBUG lava-logs => PING(20) 2019-11-05 05:28:40,745 INFO scheduling health checks: 2019-11-05 05:28:40,748 INFO scheduling jobs: 2019-11-05 05:28:41,272 DEBUG Dhanu => PING(20) 2019-11-05 05:28:47,140 DEBUG lava-logs => PING(20) 2019-11-05 05:29:00,762 INFO scheduling health checks: 2019-11-05 05:29:00,767 INFO scheduling jobs: 2019-11-05 05:29:01,293 DEBUG Dhanu => PING(20) 2019-11-05 05:29:07,157 DEBUG lava-logs => PING(20) 2019-11-05 05:29:20,781 INFO scheduling health checks: 2019-11-05 05:29:20,784 INFO scheduling jobs: 2019-11-05 05:29:21,309 DEBUG Dhanu => PING(20) 2019-11-05 05:29:27,177 DEBUG lava-logs => PING(20) 2019-11-05 05:29:40,801 INFO scheduling health checks: 2019-11-05 05:29:40,807 INFO scheduling jobs: 2019-11-05 05:29:41,329 DEBUG Dhanu => PING(20) 2019-11-05 05:29:47,198 DEBUG lava-logs => PING(20) 2019-11-05 05:30:00,825 INFO scheduling health checks: 2019-11-05 05:30:00,827 INFO scheduling jobs: 2019-11-05 05:30:01,349 DEBUG Dhanu => PING(20) 2019-11-05 05:30:07,217 DEBUG lava-logs => PING(20) 2019-11-05 05:30:20,866 INFO scheduling health checks: 2019-11-05 05:30:20,872 INFO scheduling jobs: 2019-11-05 05:30:21,369 DEBUG Dhanu => PING(20) 2019-11-05 05:30:27,238 DEBUG lava-logs => PING(20) 2019-11-05 05:30:40,889 INFO scheduling health checks: 2019-11-05 05:30:40,895 INFO scheduling jobs: 2019-11-05 05:30:41,387 DEBUG Dhanu => PING(20) 2019-11-05 05:30:47,252 DEBUG lava-logs => PING(20) 2019-11-05 05:31:00,912 INFO scheduling health checks: 2019-11-05 05:31:00,916 INFO scheduling jobs: 2019-11-05 05:31:01,411 DEBUG Dhanu => PING(20) 2019-11-05 05:31:07,274 DEBUG lava-logs => PING(20) 2019-11-05 05:31:20,934 INFO scheduling health checks: 2019-11-05 05:31:20,938 INFO scheduling jobs: 2019-11-05 05:31:21,429 DEBUG Dhanu => PING(20) 2019-11-05 05:31:27,293 DEBUG lava-logs => PING(20) 2019-11-05 05:31:40,951 INFO scheduling health checks: 2019-11-05 05:31:40,955 INFO scheduling jobs: 2019-11-05 05:31:41,450 DEBUG Dhanu => PING(20) 2019-11-05 05:31:47,316 DEBUG lava-logs => PING(20) 2019-11-05 05:32:00,970 INFO scheduling health checks: 2019-11-05 05:32:00,974 INFO scheduling jobs: 2019-11-05 05:32:01,474 DEBUG Dhanu => PING(20) 2019-11-05 05:32:07,339 DEBUG lava-logs => PING(20) 2019-11-05 05:32:20,991 INFO scheduling health checks: 2019-11-05 05:32:20,995 INFO scheduling jobs: 2019-11-05 05:32:21,489 DEBUG Dhanu => PING(20) 2019-11-05 05:32:27,361 DEBUG lava-logs => PING(20) 2019-11-05 05:32:41,012 INFO scheduling health checks: 2019-11-05 05:32:41,016 INFO scheduling jobs: 2019-11-05 05:32:41,510 DEBUG Dhanu => PING(20) 2019-11-05 05:32:47,381 DEBUG lava-logs => PING(20) 2019-11-05 05:33:01,032 INFO scheduling health checks: 2019-11-05 05:33:01,036 INFO scheduling jobs: 2019-11-05 05:33:01,534 DEBUG Dhanu => PING(20) 2019-11-05 05:33:07,391 DEBUG lava-logs => PING(20) 2019-11-05 05:33:21,054 INFO scheduling health checks: 2019-11-05 05:33:21,058 INFO scheduling jobs: 2019-11-05 05:33:21,549 DEBUG Dhanu => PING(20) 2019-11-05 05:33:27,413 DEBUG lava-logs => PING(20) 2019-11-05 05:33:41,078 INFO scheduling health checks: 2019-11-05 05:33:41,082 INFO scheduling jobs: 2019-11-05 05:33:41,569 DEBUG Dhanu => PING(20) 2019-11-05 05:33:47,432 DEBUG lava-logs => PING(20) 2019-11-05 05:34:01,095 INFO scheduling health checks: 2019-11-05 05:34:01,099 INFO scheduling jobs: 2019-11-05 05:34:01,590 DEBUG Dhanu => PING(20) 2019-11-05 05:34:07,455 DEBUG lava-logs => PING(20) 2019-11-05 05:34:21,116 INFO scheduling health checks: 2019-11-05 05:34:21,121 INFO scheduling jobs: 2019-11-05 05:34:21,614 DEBUG Dhanu => PING(20) 2019-11-05 05:34:23,108 INFO [POLL] Received a signal, leaving 2019-11-05 05:34:23,109 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 05:34:31,747 INFO [INIT] Dropping privileges 2019-11-05 05:34:31,748 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 05:34:31,748 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 05:34:31,749 INFO [INIT] Marking all workers as offline 2019-11-05 05:34:31,777 INFO [INIT] Starting encryption 2019-11-05 05:34:31,778 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 05:34:31,779 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:34:31,779 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:34:31,794 INFO [INIT] LAVA master has started. 2019-11-05 05:34:31,794 INFO [INIT] Using protocol version 3 2019-11-05 05:34:41,633 DEBUG Dhanu => PING(20) 2019-11-05 05:34:41,634 WARNING Unknown dispatcher (server crashed) 2019-11-05 05:34:46,173 DEBUG lava-logs => PING(20) 2019-11-05 05:34:46,199 DEBUG lava-logs => PING(20) 2019-11-05 05:34:51,799 INFO scheduling health checks: 2019-11-05 05:34:51,806 INFO scheduling jobs: 2019-11-05 05:35:01,673 DEBUG Dhanu => PING(20) 2019-11-05 05:35:06,267 DEBUG lava-logs => PING(20) 2019-11-05 05:35:11,820 INFO scheduling health checks: 2019-11-05 05:35:11,824 INFO scheduling jobs: 2019-11-05 05:35:21,694 DEBUG Dhanu => PING(20) 2019-11-05 05:35:26,290 DEBUG lava-logs => PING(20) 2019-11-05 05:35:31,839 INFO scheduling health checks: 2019-11-05 05:35:31,843 INFO scheduling jobs: 2019-11-05 05:35:41,717 DEBUG Dhanu => PING(20) 2019-11-05 05:35:46,305 DEBUG lava-logs => PING(20) 2019-11-05 05:35:51,857 INFO scheduling health checks: 2019-11-05 05:35:51,861 INFO scheduling jobs: 2019-11-05 05:36:01,737 DEBUG Dhanu => PING(20) 2019-11-05 05:36:06,356 DEBUG lava-logs => PING(20) 2019-11-05 05:36:11,878 INFO scheduling health checks: 2019-11-05 05:36:11,882 INFO scheduling jobs: 2019-11-05 05:36:21,762 DEBUG Dhanu => PING(20) 2019-11-05 05:36:26,379 DEBUG lava-logs => PING(20) 2019-11-05 05:36:31,898 INFO scheduling health checks: 2019-11-05 05:36:31,901 INFO scheduling jobs: 2019-11-05 05:36:41,783 DEBUG Dhanu => PING(20) 2019-11-05 05:36:46,400 DEBUG lava-logs => PING(20) 2019-11-05 05:36:51,918 INFO scheduling health checks: 2019-11-05 05:36:51,922 INFO scheduling jobs: 2019-11-05 05:37:01,805 DEBUG Dhanu => PING(20) 2019-11-05 05:37:06,422 DEBUG lava-logs => PING(20) 2019-11-05 05:37:11,936 INFO scheduling health checks: 2019-11-05 05:37:11,940 INFO scheduling jobs: 2019-11-05 05:37:21,852 DEBUG Dhanu => PING(20) 2019-11-05 05:37:26,452 DEBUG lava-logs => PING(20) 2019-11-05 05:37:31,954 INFO scheduling health checks: 2019-11-05 05:37:31,958 INFO scheduling jobs: 2019-11-05 05:37:41,873 DEBUG Dhanu => PING(20) 2019-11-05 05:37:46,473 DEBUG lava-logs => PING(20) 2019-11-05 05:37:51,972 INFO scheduling health checks: 2019-11-05 05:37:51,974 INFO scheduling jobs: 2019-11-05 05:38:01,891 DEBUG Dhanu => PING(20) 2019-11-05 05:38:06,496 DEBUG lava-logs => PING(20) 2019-11-05 05:38:11,984 INFO scheduling health checks: 2019-11-05 05:38:11,988 INFO scheduling jobs: 2019-11-05 05:38:21,911 DEBUG Dhanu => PING(20) 2019-11-05 05:38:26,513 DEBUG lava-logs => PING(20) 2019-11-05 05:38:32,003 INFO scheduling health checks: 2019-11-05 05:38:32,007 INFO scheduling jobs: 2019-11-05 05:38:41,933 DEBUG Dhanu => PING(20) 2019-11-05 05:38:46,528 DEBUG lava-logs => PING(20) 2019-11-05 05:38:52,021 INFO scheduling health checks: 2019-11-05 05:38:52,025 INFO scheduling jobs: 2019-11-05 05:39:01,953 DEBUG Dhanu => PING(20) 2019-11-05 05:39:06,541 DEBUG lava-logs => PING(20) 2019-11-05 05:39:12,044 INFO scheduling health checks: 2019-11-05 05:39:12,046 INFO scheduling jobs: 2019-11-05 05:39:21,975 DEBUG Dhanu => PING(20) 2019-11-05 05:39:26,564 DEBUG lava-logs => PING(20) 2019-11-05 05:39:31,292 INFO [POLL] Received a signal, leaving 2019-11-05 05:39:31,292 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 05:39:43,164 INFO [INIT] Dropping privileges 2019-11-05 05:39:43,164 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 05:39:43,164 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 05:39:43,166 INFO [INIT] Marking all workers as offline 2019-11-05 05:39:43,206 INFO [INIT] Starting encryption 2019-11-05 05:39:43,207 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 05:39:43,228 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:39:43,228 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:39:43,311 INFO [INIT] LAVA master has started. 2019-11-05 05:39:43,312 INFO [INIT] Using protocol version 3 2019-11-05 05:39:43,432 DEBUG Dhanu => PING(20) 2019-11-05 05:39:43,432 WARNING Unknown dispatcher (server crashed) 2019-11-05 05:39:46,579 DEBUG lava-logs => PING(20) 2019-11-05 05:39:46,623 INFO [POLL] Received a signal, leaving 2019-11-05 05:39:46,624 INFO [CLOSE] Closing the controler socket and dropping messages 2019-11-05 05:39:48,245 INFO [INIT] Dropping privileges 2019-11-05 05:39:48,245 DEBUG Switching to (lavaserver(128), lavaserver(134)) 2019-11-05 05:39:48,245 DEBUG [INIT] Dumping config to /var/lib/lava-server/default/media/lava-master-config.yaml 2019-11-05 05:39:48,247 INFO [INIT] Marking all workers as offline 2019-11-05 05:39:48,274 INFO [INIT] Starting encryption 2019-11-05 05:39:48,275 DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret 2019-11-05 05:39:48,276 DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:39:48,276 DEBUG [INIT] Watching /etc/lava-dispatcher/certificates.d/ 2019-11-05 05:39:48,293 INFO [INIT] LAVA master has started. 2019-11-05 05:39:48,293 INFO [INIT] Using protocol version 3 2019-11-05 05:40:03,448 DEBUG Dhanu => PING(20) 2019-11-05 05:40:03,448 WARNING Unknown dispatcher (server crashed) 2019-11-05 05:40:06,600 DEBUG lava-logs => PING(20) 2019-11-05 05:40:08,295 INFO scheduling health checks: 2019-11-05 05:40:08,302 INFO scheduling jobs: 2019-11-05 05:40:23,473 DEBUG Dhanu => PING(20) 2019-11-05 05:40:26,620 DEBUG lava-logs => PING(20) 2019-11-05 05:40:28,315 INFO scheduling health checks: 2019-11-05 05:40:28,319 INFO scheduling jobs: 2019-11-05 05:40:43,494 DEBUG Dhanu => PING(20) 2019-11-05 05:40:46,637 DEBUG lava-logs => PING(20) 2019-11-05 05:40:48,334 INFO scheduling health checks: 2019-11-05 05:40:48,336 INFO scheduling jobs: 2019-11-05 05:41:03,514 DEBUG Dhanu => PING(20) 2019-11-05 05:41:06,661 DEBUG lava-logs => PING(20) 2019-11-05 05:41:08,343 INFO scheduling health checks: 2019-11-05 05:41:08,354 INFO scheduling jobs: 2019-11-05 05:41:23,536 DEBUG Dhanu => PING(20) 2019-11-05 05:41:26,684 DEBUG lava-logs => PING(20) 2019-11-05 05:41:28,364 INFO scheduling health checks: 2019-11-05 05:41:28,372 INFO scheduling jobs: 2019-11-05 05:41:43,557 DEBUG Dhanu => PING(20) 2019-11-05 05:41:46,701 DEBUG lava-logs => PING(20) 2019-11-05 05:41:48,386 INFO scheduling health checks: 2019-11-05 05:41:48,394 INFO scheduling jobs: 2019-11-05 05:42:03,575 DEBUG Dhanu => PING(20) 2019-11-05 05:42:06,724 DEBUG lava-logs => PING(20) 2019-11-05 05:42:08,406 INFO scheduling health checks: 2019-11-05 05:42:08,413 INFO scheduling jobs: 2019-11-05 05:42:23,594 DEBUG Dhanu => PING(20) 2019-11-05 05:42:26,745 DEBUG lava-logs => PING(20) 2019-11-05 05:42:28,424 INFO scheduling health checks: 2019-11-05 05:42:28,436 INFO scheduling jobs: 2019-11-05 05:42:43,613 DEBUG Dhanu => PING(20) 2019-11-05 05:42:46,759 DEBUG lava-logs => PING(20) 2019-11-05 05:42:48,454 INFO scheduling health checks: 2019-11-05 05:42:48,462 INFO scheduling jobs: 2019-11-05 05:43:03,629 DEBUG Dhanu => PING(20) 2019-11-05 05:43:06,779 DEBUG lava-logs => PING(20) 2019-11-05 05:43:08,476 INFO scheduling health checks: 2019-11-05 05:43:08,485 INFO scheduling jobs: 2019-11-05 05:43:23,649 DEBUG Dhanu => PING(20) 2019-11-05 05:43:26,797 DEBUG lava-logs => PING(20) 2019-11-05 05:43:28,500 INFO scheduling health checks: 2019-11-05 05:43:28,504 INFO scheduling jobs: 2019-11-05 05:43:43,670 DEBUG Dhanu => PING(20) 2019-11-05 05:43:46,814 DEBUG lava-logs => PING(20) 2019-11-05 05:43:48,518 INFO scheduling health checks: 2019-11-05 05:43:48,522 INFO scheduling jobs: 2019-11-05 05:44:03,694 DEBUG Dhanu => PING(20) 2019-11-05 05:44:06,838 DEBUG lava-logs => PING(20) 2019-11-05 05:44:08,536 INFO scheduling health checks: 2019-11-05 05:44:08,540 INFO scheduling jobs: 2019-11-05 05:44:23,713 DEBUG Dhanu => PING(20) 2019-11-05 05:44:26,857 DEBUG lava-logs => PING(20) 2019-11-05 05:44:28,555 INFO scheduling health checks: 2019-11-05 05:44:28,566 INFO scheduling jobs: 2019-11-05 05:44:43,731 DEBUG Dhanu => PING(20) 2019-11-05 05:44:46,877 DEBUG lava-logs => PING(20) 2019-11-05 05:44:48,580 INFO scheduling health checks: 2019-11-05 05:44:48,591 INFO scheduling jobs: 2019-11-05 05:45:03,749 DEBUG Dhanu => PING(20) 2019-11-05 05:45:06,898 DEBUG lava-logs => PING(20) 2019-11-05 05:45:08,601 INFO scheduling health checks: 2019-11-05 05:45:08,638 INFO scheduling jobs: 2019-11-05 05:45:08,640 DEBUG - qemu 2019-11-05 05:45:23,768 DEBUG Dhanu => PING(20) 2019-11-05 05:45:26,917 DEBUG lava-logs => PING(20) 2019-11-05 05:45:28,671 INFO scheduling health checks: 2019-11-05 05:45:28,718 INFO scheduling jobs: 2019-11-05 05:45:28,721 DEBUG - qemu 2019-11-05 05:45:43,792 DEBUG Dhanu => PING(20) 2019-11-05 05:45:46,937 DEBUG lava-logs => PING(20) 2019-11-05 05:45:48,749 INFO scheduling health checks: 2019-11-05 05:45:48,764 INFO scheduling jobs: 2019-11-05 05:45:48,766 DEBUG - qemu 2019-11-05 05:46:03,809 DEBUG Dhanu => PING(20) 2019-11-05 05:46:06,959 DEBUG lava-logs => PING(20) 2019-11-05 05:46:08,788 INFO scheduling health checks: 2019-11-05 05:46:08,822 INFO scheduling jobs: 2019-11-05 05:46:08,825 DEBUG - qemu 2019-11-05 05:46:23,827 DEBUG Dhanu => PING(20) 2019-11-05 05:46:26,978 DEBUG lava-logs => PING(20) 2019-11-05 05:46:28,857 INFO scheduling health checks: 2019-11-05 05:46:28,890 INFO scheduling jobs: 2019-11-05 05:46:28,893 DEBUG - qemu 2019-11-05 05:46:43,847 DEBUG Dhanu => PING(20) 2019-11-05 05:46:46,997 DEBUG lava-logs => PING(20) 2019-11-05 05:46:48,931 INFO scheduling health checks: 2019-11-05 05:46:48,948 INFO scheduling jobs: 2019-11-05 05:46:48,951 DEBUG - qemu 2019-11-05 05:47:03,870 DEBUG Dhanu => PING(20) 2019-11-05 05:47:07,020 DEBUG lava-logs => PING(20) 2019-11-05 05:47:08,983 INFO scheduling health checks: 2019-11-05 05:47:09,028 INFO scheduling jobs: 2019-11-05 05:47:09,030 DEBUG - qemu 2019-11-05 05:47:23,893 DEBUG Dhanu => PING(20) 2019-11-05 05:47:27,041 DEBUG lava-logs => PING(20) 2019-11-05 05:47:29,063 INFO scheduling health checks: 2019-11-05 05:47:29,076 DEBUG - qemu 2019-11-05 05:47:29,076 DEBUG -> qemu01 (Idle, Good) 2019-11-05 05:47:29,136 DEBUG |--> scheduling health check 2019-11-05 05:47:29,196 INFO scheduling jobs: 2019-11-05 05:47:29,334 INFO [29] START => Dhanu (qemu01) 2019-11-05 05:47:29,557 INFO [29] Dhanu => START_OK 2019-11-05 05:47:29,663 INFO scheduling health checks: 2019-11-05 05:47:29,668 INFO scheduling jobs: 2019-11-05 05:47:34,629 INFO [29] Dhanu => END 2019-11-05 05:47:35,753 DEBUG get_type_name: skipping finalize 2019-11-05 05:47:35,753 DEBUG type_name failed for finalize metatype finalize 2019-11-05 05:47:35,753 DEBUG get_type_name: skipping finalize 2019-11-05 05:47:35,754 DEBUG type_name failed for finalize metatype finalize 2019-11-05 05:47:35,754 DEBUG get_type_name: skipping finalize 2019-11-05 05:47:35,754 DEBUG type_name failed for finalize metatype finalize 2019-11-05 05:47:49,377 INFO scheduling health checks: 2019-11-05 05:47:49,387 INFO scheduling jobs: 2019-11-05 05:47:50,913 DEBUG Dhanu => PING(20) 2019-11-05 05:47:54,609 DEBUG lava-logs => PING(20) 2019-11-05 05:48:09,406 INFO scheduling health checks: 2019-11-05 05:48:09,416 INFO scheduling jobs: 2019-11-05 05:48:10,933 DEBUG Dhanu => PING(20) 2019-11-05 05:48:14,629 DEBUG lava-logs => PING(20) 2019-11-05 05:48:29,429 INFO scheduling health checks: 2019-11-05 05:48:29,440 INFO scheduling jobs: 2019-11-05 05:48:30,950 DEBUG Dhanu => PING(20) 2019-11-05 05:48:34,649 DEBUG lava-logs => PING(20) 2019-11-05 05:48:49,456 INFO scheduling health checks: 2019-11-05 05:48:49,467 INFO scheduling jobs: 2019-11-05 05:48:50,974 DEBUG Dhanu => PING(20) 2019-11-05 05:48:54,666 DEBUG lava-logs => PING(20) 2019-11-05 05:49:09,478 INFO scheduling health checks: 2019-11-05 05:49:09,488 INFO scheduling jobs: 2019-11-05 05:49:10,993 DEBUG Dhanu => PING(20) 2019-11-05 05:49:14,685 DEBUG lava-logs => PING(20) 2019-11-05 05:49:29,501 INFO scheduling health checks: 2019-11-05 05:49:29,512 INFO scheduling jobs: 2019-11-05 05:49:31,009 DEBUG Dhanu => PING(20) 2019-11-05 05:49:34,701 DEBUG lava-logs => PING(20) 2019-11-05 05:49:49,524 INFO scheduling health checks: 2019-11-05 05:49:49,530 INFO scheduling jobs: 2019-11-05 05:49:51,025 DEBUG Dhanu => PING(20) 2019-11-05 05:49:54,723 DEBUG lava-logs => PING(20) 2019-11-05 05:50:09,541 INFO scheduling health checks: 2019-11-05 05:50:09,554 INFO scheduling jobs: 2019-11-05 05:50:11,044 DEBUG Dhanu => PING(20) 2019-11-05 05:50:14,741 DEBUG lava-logs => PING(20) 2019-11-05 05:50:29,568 INFO scheduling health checks: 2019-11-05 05:50:29,579 INFO scheduling jobs: 2019-11-05 05:50:31,064 DEBUG Dhanu => PING(20) 2019-11-05 05:50:34,761 DEBUG lava-logs => PING(20) 2019-11-05 05:50:49,588 INFO scheduling health checks: 2019-11-05 05:50:49,600 INFO scheduling jobs: 2019-11-05 05:50:51,081 DEBUG Dhanu => PING(20) 2019-11-05 05:50:54,783 DEBUG lava-logs => PING(20) 2019-11-05 05:51:09,612 INFO scheduling health checks: 2019-11-05 05:51:09,622 INFO scheduling jobs: 2019-11-05 05:51:11,098 DEBUG Dhanu => PING(20) 2019-11-05 05:51:14,801 DEBUG lava-logs => PING(20) 2019-11-05 05:51:29,633 INFO scheduling health checks: 2019-11-05 05:51:29,644 INFO scheduling jobs: 2019-11-05 05:51:31,117 DEBUG Dhanu => PING(20) 2019-11-05 05:51:34,823 DEBUG lava-logs => PING(20) 2019-11-05 05:51:49,655 INFO scheduling health checks: 2019-11-05 05:51:49,663 INFO scheduling jobs: 2019-11-05 05:51:51,137 DEBUG Dhanu => PING(20) 2019-11-05 05:51:54,842 DEBUG lava-logs => PING(20) 2019-11-05 05:52:09,676 INFO scheduling health checks: 2019-11-05 05:52:09,687 INFO scheduling jobs: 2019-11-05 05:52:11,154 DEBUG Dhanu => PING(20) 2019-11-05 05:52:14,864 DEBUG lava-logs => PING(20) 2019-11-05 05:52:29,704 INFO scheduling health checks: 2019-11-05 05:52:29,714 INFO scheduling jobs: 2019-11-05 05:52:31,167 DEBUG Dhanu => PING(20) 2019-11-05 05:52:34,881 DEBUG lava-logs => PING(20) 2019-11-05 05:52:49,724 INFO scheduling health checks: 2019-11-05 05:52:49,736 INFO scheduling jobs: 2019-11-05 05:52:51,185 DEBUG Dhanu => PING(20) 2019-11-05 05:52:54,895 DEBUG lava-logs => PING(20) 2019-11-05 05:53:09,748 INFO scheduling health checks: 2019-11-05 05:53:09,758 INFO scheduling jobs: 2019-11-05 05:53:11,205 DEBUG Dhanu => PING(20) 2019-11-05 05:53:14,918 DEBUG lava-logs => PING(20) 2019-11-05 05:53:29,770 INFO scheduling health checks: 2019-11-05 05:53:29,781 INFO scheduling jobs: 2019-11-05 05:53:31,227 DEBUG Dhanu => PING(20) 2019-11-05 05:53:34,942 DEBUG lava-logs => PING(20) 2019-11-05 05:53:49,792 INFO scheduling health checks: 2019-11-05 05:53:49,800 INFO scheduling jobs: 2019-11-05 05:53:51,246 DEBUG Dhanu => PING(20) 2019-11-05 05:53:54,961 DEBUG lava-logs => PING(20) 2019-11-05 05:54:09,815 INFO scheduling health checks: 2019-11-05 05:54:09,823 INFO scheduling jobs: 2019-11-05 05:54:11,265 DEBUG Dhanu => PING(20) 2019-11-05 05:54:14,982 DEBUG lava-logs => PING(20) 2019-11-05 05:54:29,839 INFO scheduling health checks: 2019-11-05 05:54:29,843 INFO scheduling jobs: 2019-11-05 05:54:31,284 DEBUG Dhanu => PING(20) 2019-11-05 05:54:34,997 DEBUG lava-logs => PING(20) 2019-11-05 05:54:49,850 INFO scheduling health checks: 2019-11-05 05:54:49,859 INFO scheduling jobs: 2019-11-05 05:54:51,302 DEBUG Dhanu => PING(20) 2019-11-05 05:54:55,018 DEBUG lava-logs => PING(20) 2019-11-05 05:55:09,874 INFO scheduling health checks: 2019-11-05 05:55:09,878 INFO scheduling jobs: 2019-11-05 05:55:11,321 DEBUG Dhanu => PING(20) 2019-11-05 05:55:15,039 DEBUG lava-logs => PING(20) 2019-11-05 05:55:29,893 INFO scheduling health checks: 2019-11-05 05:55:29,903 INFO scheduling jobs: 2019-11-05 05:55:31,342 DEBUG Dhanu => PING(20) 2019-11-05 05:55:35,057 DEBUG lava-logs => PING(20) 2019-11-05 05:55:49,919 INFO scheduling health checks: 2019-11-05 05:55:49,930 INFO scheduling jobs: 2019-11-05 05:55:51,361 DEBUG Dhanu => PING(20) 2019-11-05 05:55:55,077 DEBUG lava-logs => PING(20) 2019-11-05 05:56:09,941 INFO scheduling health checks: 2019-11-05 05:56:09,950 INFO scheduling jobs: 2019-11-05 05:56:11,382 DEBUG Dhanu => PING(20) 2019-11-05 05:56:15,098 DEBUG lava-logs => PING(20) 2019-11-05 05:56:29,963 INFO scheduling health checks: 2019-11-05 05:56:29,974 INFO scheduling jobs: 2019-11-05 05:56:31,401 DEBUG Dhanu => PING(20) 2019-11-05 05:56:35,136 DEBUG lava-logs => PING(20) 2019-11-05 05:56:49,984 INFO scheduling health checks: 2019-11-05 05:56:49,997 INFO scheduling jobs: 2019-11-05 05:56:51,422 DEBUG Dhanu => PING(20) 2019-11-05 05:56:55,157 DEBUG lava-logs => PING(20) 2019-11-05 05:57:10,010 INFO scheduling health checks: 2019-11-05 05:57:10,017 INFO scheduling jobs: 2019-11-05 05:57:11,437 DEBUG Dhanu => PING(20) 2019-11-05 05:57:15,180 DEBUG lava-logs => PING(20) 2019-11-05 05:57:30,028 INFO scheduling health checks: 2019-11-05 05:57:30,038 INFO scheduling jobs: 2019-11-05 05:57:31,456 DEBUG Dhanu => PING(20) 2019-11-05 05:57:35,201 DEBUG lava-logs => PING(20) 2019-11-05 05:57:50,050 INFO scheduling health checks: 2019-11-05 05:57:50,058 INFO scheduling jobs: 2019-11-05 05:57:51,505 DEBUG Dhanu => PING(20) 2019-11-05 05:57:55,216 DEBUG lava-logs => PING(20) 2019-11-05 05:58:10,068 INFO scheduling health checks: 2019-11-05 05:58:10,075 INFO scheduling jobs: 2019-11-05 05:58:11,527 DEBUG Dhanu => PING(20) 2019-11-05 05:58:15,239 DEBUG lava-logs => PING(20) 2019-11-05 05:58:30,090 INFO scheduling health checks: 2019-11-05 05:58:30,101 INFO scheduling jobs: 2019-11-05 05:58:31,548 DEBUG Dhanu => PING(20) 2019-11-05 05:58:35,261 DEBUG lava-logs => PING(20) 2019-11-05 05:58:50,111 INFO scheduling health checks: 2019-11-05 05:58:50,120 INFO scheduling jobs: 2019-11-05 05:58:51,565 DEBUG Dhanu => PING(20) 2019-11-05 05:58:55,281 DEBUG lava-logs => PING(20) 2019-11-05 05:59:10,132 INFO scheduling health checks: 2019-11-05 05:59:10,144 INFO scheduling jobs: 2019-11-05 05:59:11,589 DEBUG Dhanu => PING(20) 2019-11-05 05:59:15,305 DEBUG lava-logs => PING(20) 2019-11-05 05:59:30,155 INFO scheduling health checks: 2019-11-05 05:59:30,160 INFO scheduling jobs: 2019-11-05 05:59:31,611 DEBUG Dhanu => PING(20) 2019-11-05 05:59:35,325 DEBUG lava-logs => PING(20) 2019-11-05 05:59:50,168 INFO scheduling health checks: 2019-11-05 05:59:50,181 INFO scheduling jobs: 2019-11-05 05:59:51,633 DEBUG Dhanu => PING(20) 2019-11-05 05:59:55,373 DEBUG lava-logs => PING(20) 2019-11-05 06:00:10,193 INFO scheduling health checks: 2019-11-05 06:00:10,204 INFO scheduling jobs: 2019-11-05 06:00:11,653 DEBUG Dhanu => PING(20) 2019-11-05 06:00:15,393 DEBUG lava-logs => PING(20) 2019-11-05 06:00:30,216 INFO scheduling health checks: 2019-11-05 06:00:30,226 INFO scheduling jobs: 2019-11-05 06:00:31,669 DEBUG Dhanu => PING(20) 2019-11-05 06:00:35,413 DEBUG lava-logs => PING(20) 2019-11-05 06:00:50,236 INFO scheduling health checks: 2019-11-05 06:00:50,248 INFO scheduling jobs: 2019-11-05 06:00:51,693 DEBUG Dhanu => PING(20) 2019-11-05 06:00:55,420 DEBUG lava-logs => PING(20) 2019-11-05 06:01:10,260 INFO scheduling health checks: 2019-11-05 06:01:10,271 INFO scheduling jobs: 2019-11-05 06:01:11,709 DEBUG Dhanu => PING(20) 2019-11-05 06:01:15,429 DEBUG lava-logs => PING(20) 2019-11-05 06:01:30,286 INFO scheduling health checks: 2019-11-05 06:01:30,298 INFO scheduling jobs: 2019-11-05 06:01:31,721 DEBUG Dhanu => PING(20) 2019-11-05 06:01:35,449 DEBUG lava-logs => PING(20) 2019-11-05 06:01:50,315 INFO scheduling health checks: 2019-11-05 06:01:50,325 INFO scheduling jobs: 2019-11-05 06:01:51,746 DEBUG Dhanu => PING(20) 2019-11-05 06:01:55,469 DEBUG lava-logs => PING(20) 2019-11-05 06:02:10,336 INFO scheduling health checks: 2019-11-05 06:02:10,348 INFO scheduling jobs: 2019-11-05 06:02:11,765 DEBUG Dhanu => PING(20) 2019-11-05 06:02:15,491 DEBUG lava-logs => PING(20) 2019-11-05 06:02:30,362 INFO scheduling health checks: 2019-11-05 06:02:30,373 INFO scheduling jobs: 2019-11-05 06:02:31,781 DEBUG Dhanu => PING(20) 2019-11-05 06:02:35,508 DEBUG lava-logs => PING(20) 2019-11-05 06:02:50,383 INFO scheduling health checks: 2019-11-05 06:02:50,394 INFO scheduling jobs: 2019-11-05 06:02:51,797 DEBUG Dhanu => PING(20) 2019-11-05 06:02:55,525 DEBUG lava-logs => PING(20) 2019-11-05 06:03:10,405 INFO scheduling health checks: 2019-11-05 06:03:10,416 INFO scheduling jobs: 2019-11-05 06:03:11,813 DEBUG Dhanu => PING(20) 2019-11-05 06:03:15,548 DEBUG lava-logs => PING(20) 2019-11-05 06:03:30,430 INFO scheduling health checks: 2019-11-05 06:03:30,440 INFO scheduling jobs: 2019-11-05 06:03:31,837 DEBUG Dhanu => PING(20) 2019-11-05 06:03:35,571 DEBUG lava-logs => PING(20) 2019-11-05 06:03:50,452 INFO scheduling health checks: 2019-11-05 06:03:50,463 INFO scheduling jobs: 2019-11-05 06:03:51,855 DEBUG Dhanu => PING(20) 2019-11-05 06:03:55,594 DEBUG lava-logs => PING(20) 2019-11-05 06:04:10,474 INFO scheduling health checks: 2019-11-05 06:04:10,485 INFO scheduling jobs: 2019-11-05 06:04:11,870 DEBUG Dhanu => PING(20) 2019-11-05 06:04:15,613 DEBUG lava-logs => PING(20) 2019-11-05 06:04:30,499 INFO scheduling health checks: 2019-11-05 06:04:30,510 INFO scheduling jobs: 2019-11-05 06:04:31,885 DEBUG Dhanu => PING(20) 2019-11-05 06:04:35,633 DEBUG lava-logs => PING(20) 2019-11-05 06:04:50,523 INFO scheduling health checks: 2019-11-05 06:04:50,531 INFO scheduling jobs: 2019-11-05 06:04:51,901 DEBUG Dhanu => PING(20) 2019-11-05 06:04:55,653 DEBUG lava-logs => PING(20) 2019-11-05 06:05:10,543 INFO scheduling health checks: 2019-11-05 06:05:10,553 INFO scheduling jobs: 2019-11-05 06:05:11,923 DEBUG Dhanu => PING(20) 2019-11-05 06:05:15,669 DEBUG lava-logs => PING(20) 2019-11-05 06:05:30,566 INFO scheduling health checks: 2019-11-05 06:05:30,575 INFO scheduling jobs: 2019-11-05 06:05:31,945 DEBUG Dhanu => PING(20) 2019-11-05 06:05:35,689 DEBUG lava-logs => PING(20) 2019-11-05 06:05:50,589 INFO scheduling health checks: 2019-11-05 06:05:50,597 INFO scheduling jobs: 2019-11-05 06:05:51,963 DEBUG Dhanu => PING(20) 2019-11-05 06:05:55,712 DEBUG lava-logs => PING(20) 2019-11-05 06:06:10,610 INFO scheduling health checks: 2019-11-05 06:06:10,623 INFO scheduling jobs: 2019-11-05 06:06:11,985 DEBUG Dhanu => PING(20) 2019-11-05 06:06:15,729 DEBUG lava-logs => PING(20) 2019-11-05 06:06:30,633 INFO scheduling health checks: 2019-11-05 06:06:30,644 INFO scheduling jobs: 2019-11-05 06:06:31,999 DEBUG Dhanu => PING(20) 2019-11-05 06:06:35,752 DEBUG lava-logs => PING(20) 2019-11-05 06:06:50,660 INFO scheduling health checks: 2019-11-05 06:06:50,671 INFO scheduling jobs: 2019-11-05 06:06:52,017 DEBUG Dhanu => PING(20) 2019-11-05 06:06:55,773 DEBUG lava-logs => PING(20) 2019-11-05 06:07:10,684 INFO scheduling health checks: 2019-11-05 06:07:10,694 INFO scheduling jobs: 2019-11-05 06:07:12,036 DEBUG Dhanu => PING(20) 2019-11-05 06:07:15,796 DEBUG lava-logs => PING(20) 2019-11-05 06:07:30,709 INFO scheduling health checks: 2019-11-05 06:07:30,722 INFO scheduling jobs: 2019-11-05 06:07:32,053 DEBUG Dhanu => PING(20) 2019-11-05 06:07:35,818 DEBUG lava-logs => PING(20) 2019-11-05 06:07:50,734 INFO scheduling health checks: 2019-11-05 06:07:50,745 INFO scheduling jobs: 2019-11-05 06:07:52,071 DEBUG Dhanu => PING(20) 2019-11-05 06:07:55,841 DEBUG lava-logs => PING(20) 2019-11-05 06:08:10,758 INFO scheduling health checks: 2019-11-05 06:08:10,768 INFO scheduling jobs: 2019-11-05 06:08:12,091 DEBUG Dhanu => PING(20) 2019-11-05 06:08:15,864 DEBUG lava-logs => PING(20) 2019-11-05 06:08:30,779 INFO scheduling health checks: 2019-11-05 06:08:30,790 INFO scheduling jobs: 2019-11-05 06:08:32,115 DEBUG Dhanu => PING(20) 2019-11-05 06:08:35,887 DEBUG lava-logs => PING(20) 2019-11-05 06:08:50,803 INFO scheduling health checks: 2019-11-05 06:08:50,812 INFO scheduling jobs: 2019-11-05 06:08:52,139 DEBUG Dhanu => PING(20) 2019-11-05 06:08:55,904 DEBUG lava-logs => PING(20) 2019-11-05 06:09:10,824 INFO scheduling health checks: 2019-11-05 06:09:10,834 INFO scheduling jobs: 2019-11-05 06:09:12,163 DEBUG Dhanu => PING(20) 2019-11-05 06:09:15,927 DEBUG lava-logs => PING(20) 2019-11-05 06:09:30,848 INFO scheduling health checks: 2019-11-05 06:09:30,857 INFO scheduling jobs: 2019-11-05 06:09:32,187 DEBUG Dhanu => PING(20) 2019-11-05 06:09:35,948 DEBUG lava-logs => PING(20) 2019-11-05 06:09:50,868 INFO scheduling health checks: 2019-11-05 06:09:50,879 INFO scheduling jobs: 2019-11-05 06:09:52,210 DEBUG Dhanu => PING(20) 2019-11-05 06:09:55,968 DEBUG lava-logs => PING(20) 2019-11-05 06:10:10,892 INFO scheduling health checks: 2019-11-05 06:10:10,903 INFO scheduling jobs: 2019-11-05 06:10:12,233 DEBUG Dhanu => PING(20) 2019-11-05 06:10:15,988 DEBUG lava-logs => PING(20) 2019-11-05 06:10:30,917 INFO scheduling health checks: 2019-11-05 06:10:30,927 INFO scheduling jobs: 2019-11-05 06:10:32,253 DEBUG Dhanu => PING(20) 2019-11-05 06:10:36,008 DEBUG lava-logs => PING(20) 2019-11-05 06:10:50,939 INFO scheduling health checks: 2019-11-05 06:10:50,951 INFO scheduling jobs: 2019-11-05 06:10:52,274 DEBUG Dhanu => PING(20) 2019-11-05 06:10:56,031 DEBUG lava-logs => PING(20) 2019-11-05 06:11:10,962 INFO scheduling health checks: 2019-11-05 06:11:10,973 INFO scheduling jobs: 2019-11-05 06:11:12,294 DEBUG Dhanu => PING(20) 2019-11-05 06:11:16,054 DEBUG lava-logs => PING(20) 2019-11-05 06:11:30,982 INFO scheduling health checks: 2019-11-05 06:11:30,993 INFO scheduling jobs: 2019-11-05 06:11:32,317 DEBUG Dhanu => PING(20) 2019-11-05 06:11:36,077 DEBUG lava-logs => PING(20) 2019-11-05 06:11:51,004 INFO scheduling health checks: 2019-11-05 06:11:51,015 INFO scheduling jobs: 2019-11-05 06:11:52,342 DEBUG Dhanu => PING(20) 2019-11-05 06:11:56,100 DEBUG lava-logs => PING(20) 2019-11-05 06:12:11,032 INFO scheduling health checks: 2019-11-05 06:12:11,043 INFO scheduling jobs: 2019-11-05 06:12:12,364 DEBUG Dhanu => PING(20) 2019-11-05 06:12:16,123 DEBUG lava-logs => PING(20) 2019-11-05 06:12:31,052 INFO scheduling health checks: 2019-11-05 06:12:31,064 INFO scheduling jobs: 2019-11-05 06:12:32,389 DEBUG Dhanu => PING(20) 2019-11-05 06:12:36,146 DEBUG lava-logs => PING(20) 2019-11-05 06:12:51,075 INFO scheduling health checks: 2019-11-05 06:12:51,087 INFO scheduling jobs: 2019-11-05 06:12:52,411 DEBUG Dhanu => PING(20) 2019-11-05 06:12:56,161 DEBUG lava-logs => PING(20) 2019-11-05 06:13:11,103 INFO scheduling health checks: 2019-11-05 06:13:11,114 INFO scheduling jobs: 2019-11-05 06:13:12,433 DEBUG Dhanu => PING(20) 2019-11-05 06:13:16,184 DEBUG lava-logs => PING(20) 2019-11-05 06:13:31,129 INFO scheduling health checks: 2019-11-05 06:13:31,139 INFO scheduling jobs: 2019-11-05 06:13:32,458 DEBUG Dhanu => PING(20) 2019-11-05 06:13:36,207 DEBUG lava-logs => PING(20) 2019-11-05 06:13:51,152 INFO scheduling health checks: 2019-11-05 06:13:51,163 INFO scheduling jobs: 2019-11-05 06:13:52,483 DEBUG Dhanu => PING(20) 2019-11-05 06:13:56,229 DEBUG lava-logs => PING(20) 2019-11-05 06:14:11,172 INFO scheduling health checks: 2019-11-05 06:14:11,182 INFO scheduling jobs: 2019-11-05 06:14:12,507 DEBUG Dhanu => PING(20) 2019-11-05 06:14:16,249 DEBUG lava-logs => PING(20) 2019-11-05 06:14:31,193 INFO scheduling health checks: 2019-11-05 06:14:31,204 INFO scheduling jobs: 2019-11-05 06:14:32,531 DEBUG Dhanu => PING(20) 2019-11-05 06:14:36,265 DEBUG lava-logs => PING(20) 2019-11-05 06:14:51,223 INFO scheduling health checks: 2019-11-05 06:14:51,232 INFO scheduling jobs: 2019-11-05 06:14:52,556 DEBUG Dhanu => PING(20) 2019-11-05 06:14:56,288 DEBUG lava-logs => PING(20) 2019-11-05 06:15:11,245 INFO scheduling health checks: 2019-11-05 06:15:11,255 INFO scheduling jobs: 2019-11-05 06:15:12,576 DEBUG Dhanu => PING(20) 2019-11-05 06:15:16,311 DEBUG lava-logs => PING(20) 2019-11-05 06:15:31,274 INFO scheduling health checks: 2019-11-05 06:15:31,285 INFO scheduling jobs: 2019-11-05 06:15:32,599 DEBUG Dhanu => PING(20) 2019-11-05 06:15:36,364 DEBUG lava-logs => PING(20) 2019-11-05 06:15:51,298 INFO scheduling health checks: 2019-11-05 06:15:51,309 INFO scheduling jobs: 2019-11-05 06:15:52,623 DEBUG Dhanu => PING(20) 2019-11-05 06:15:56,385 DEBUG lava-logs => PING(20) 2019-11-05 06:16:11,323 INFO scheduling health checks: 2019-11-05 06:16:11,335 INFO scheduling jobs: 2019-11-05 06:16:12,645 DEBUG Dhanu => PING(20) 2019-11-05 06:16:16,407 DEBUG lava-logs => PING(20) 2019-11-05 06:16:31,346 INFO scheduling health checks: 2019-11-05 06:16:31,357 INFO scheduling jobs: 2019-11-05 06:16:32,669 DEBUG Dhanu => PING(20) 2019-11-05 06:16:36,423 DEBUG lava-logs => PING(20) 2019-11-05 06:16:51,369 INFO scheduling health checks: 2019-11-05 06:16:51,379 INFO scheduling jobs: 2019-11-05 06:16:52,690 DEBUG Dhanu => PING(20) 2019-11-05 06:16:56,443 DEBUG lava-logs => PING(20) 2019-11-05 06:17:11,393 INFO scheduling health checks: 2019-11-05 06:17:11,404 INFO scheduling jobs: 2019-11-05 06:17:12,713 DEBUG Dhanu => PING(20) 2019-11-05 06:17:16,465 DEBUG lava-logs => PING(20) 2019-11-05 06:17:31,415 INFO scheduling health checks: 2019-11-05 06:17:31,426 INFO scheduling jobs: 2019-11-05 06:17:32,732 DEBUG Dhanu => PING(20) 2019-11-05 06:17:36,488 DEBUG lava-logs => PING(20) 2019-11-05 06:17:51,437 INFO scheduling health checks: 2019-11-05 06:17:51,446 INFO scheduling jobs: 2019-11-05 06:17:52,756 DEBUG Dhanu => PING(20) 2019-11-05 06:17:56,504 DEBUG lava-logs => PING(20) 2019-11-05 06:18:11,464 INFO scheduling health checks: 2019-11-05 06:18:11,468 INFO scheduling jobs: 2019-11-05 06:18:12,772 DEBUG Dhanu => PING(20) 2019-11-05 06:18:16,524 DEBUG lava-logs => PING(20) 2019-11-05 06:18:31,476 INFO scheduling health checks: 2019-11-05 06:18:31,481 INFO scheduling jobs: 2019-11-05 06:18:32,796 DEBUG Dhanu => PING(20) 2019-11-05 06:18:36,540 DEBUG lava-logs => PING(20) 2019-11-05 06:18:51,492 INFO scheduling health checks: 2019-11-05 06:18:51,502 INFO scheduling jobs: 2019-11-05 06:18:52,820 DEBUG Dhanu => PING(20) 2019-11-05 06:18:56,561 DEBUG lava-logs => PING(20) 2019-11-05 06:19:11,512 INFO scheduling health checks: 2019-11-05 06:19:11,524 INFO scheduling jobs: 2019-11-05 06:19:12,839 DEBUG Dhanu => PING(20) 2019-11-05 06:19:16,584 DEBUG lava-logs => PING(20) 2019-11-05 06:19:31,540 INFO scheduling health checks: 2019-11-05 06:19:31,550 INFO scheduling jobs: 2019-11-05 06:19:32,861 DEBUG Dhanu => PING(20) 2019-11-05 06:19:36,607 DEBUG lava-logs => PING(20) 2019-11-05 06:19:51,564 INFO scheduling health checks: 2019-11-05 06:19:51,575 INFO scheduling jobs: 2019-11-05 06:19:52,885 DEBUG Dhanu => PING(20) 2019-11-05 06:19:56,630 DEBUG lava-logs => PING(20) 2019-11-05 06:20:11,591 INFO scheduling health checks: 2019-11-05 06:20:11,599 INFO scheduling jobs: 2019-11-05 06:20:12,906 DEBUG Dhanu => PING(20) 2019-11-05 06:20:16,652 DEBUG lava-logs => PING(20) 2019-11-05 06:20:31,616 INFO scheduling health checks: 2019-11-05 06:20:31,626 INFO scheduling jobs: 2019-11-05 06:20:32,926 DEBUG Dhanu => PING(20) 2019-11-05 06:20:36,674 DEBUG lava-logs => PING(20) 2019-11-05 06:20:51,644 INFO scheduling health checks: 2019-11-05 06:20:51,649 INFO scheduling jobs: 2019-11-05 06:20:52,945 DEBUG Dhanu => PING(20) 2019-11-05 06:20:56,696 DEBUG lava-logs => PING(20) 2019-11-05 06:21:11,666 INFO scheduling health checks: 2019-11-05 06:21:11,676 INFO scheduling jobs: 2019-11-05 06:21:12,961 DEBUG Dhanu => PING(20) 2019-11-05 06:21:16,719 DEBUG lava-logs => PING(20) 2019-11-05 06:21:31,692 INFO scheduling health checks: 2019-11-05 06:21:31,702 INFO scheduling jobs: 2019-11-05 06:21:32,982 DEBUG Dhanu => PING(20) 2019-11-05 06:21:36,742 DEBUG lava-logs => PING(20) 2019-11-05 06:21:51,715 INFO scheduling health checks: 2019-11-05 06:21:51,725 INFO scheduling jobs: 2019-11-05 06:21:53,007 DEBUG Dhanu => PING(20) 2019-11-05 06:21:56,761 DEBUG lava-logs => PING(20) 2019-11-05 06:22:11,739 INFO scheduling health checks: 2019-11-05 06:22:11,750 INFO scheduling jobs: 2019-11-05 06:22:13,025 DEBUG Dhanu => PING(20) 2019-11-05 06:22:16,781 DEBUG lava-logs => PING(20) 2019-11-05 06:22:31,768 INFO scheduling health checks: 2019-11-05 06:22:31,774 INFO scheduling jobs: 2019-11-05 06:22:33,046 DEBUG Dhanu => PING(20) 2019-11-05 06:22:36,797 DEBUG lava-logs => PING(20) 2019-11-05 06:22:51,788 INFO scheduling health checks: 2019-11-05 06:22:51,799 INFO scheduling jobs: 2019-11-05 06:22:53,065 DEBUG Dhanu => PING(20) 2019-11-05 06:22:56,809 DEBUG lava-logs => PING(20) 2019-11-05 06:23:11,812 INFO scheduling health checks: 2019-11-05 06:23:11,822 INFO scheduling jobs: 2019-11-05 06:23:13,086 DEBUG Dhanu => PING(20) 2019-11-05 06:23:16,829 DEBUG lava-logs => PING(20) 2019-11-05 06:23:31,835 INFO scheduling health checks: 2019-11-05 06:23:31,844 INFO scheduling jobs: 2019-11-05 06:23:33,107 DEBUG Dhanu => PING(20) 2019-11-05 06:23:36,843 DEBUG lava-logs => PING(20) 2019-11-05 06:23:51,858 INFO scheduling health checks: 2019-11-05 06:23:51,869 INFO scheduling jobs: 2019-11-05 06:23:53,125 DEBUG Dhanu => PING(20) 2019-11-05 06:23:56,861 DEBUG lava-logs => PING(20) 2019-11-05 06:24:11,880 INFO scheduling health checks: 2019-11-05 06:24:11,886 INFO scheduling jobs: 2019-11-05 06:24:13,145 DEBUG Dhanu => PING(20) 2019-11-05 06:24:16,882 DEBUG lava-logs => PING(20) 2019-11-05 06:24:31,900 INFO scheduling health checks: 2019-11-05 06:24:31,911 INFO scheduling jobs: 2019-11-05 06:24:33,167 DEBUG Dhanu => PING(20) 2019-11-05 06:24:36,901 DEBUG lava-logs => PING(20) 2019-11-05 06:24:51,923 INFO scheduling health checks: 2019-11-05 06:24:51,931 INFO scheduling jobs: 2019-11-05 06:24:53,188 DEBUG Dhanu => PING(20) 2019-11-05 06:24:56,924 DEBUG lava-logs => PING(20) 2019-11-05 06:25:11,944 INFO scheduling health checks: 2019-11-05 06:25:11,956 INFO scheduling jobs: 2019-11-05 06:25:13,205 DEBUG Dhanu => PING(20) 2019-11-05 06:25:16,943 DEBUG lava-logs => PING(20) 2019-11-05 06:25:31,976 INFO scheduling health checks: 2019-11-05 06:25:31,986 INFO scheduling jobs: 2019-11-05 06:25:33,229 DEBUG Dhanu => PING(20) 2019-11-05 06:25:36,965 DEBUG lava-logs => PING(20) 2019-11-05 06:25:51,999 INFO scheduling health checks: 2019-11-05 06:25:52,009 INFO scheduling jobs: 2019-11-05 06:25:53,249 DEBUG Dhanu => PING(20) 2019-11-05 06:25:57,005 DEBUG lava-logs => PING(20) 2019-11-05 06:26:12,020 INFO scheduling health checks: 2019-11-05 06:26:12,031 INFO scheduling jobs: 2019-11-05 06:26:13,266 DEBUG Dhanu => PING(20) 2019-11-05 06:26:17,021 DEBUG lava-logs => PING(20) 2019-11-05 06:26:32,048 INFO scheduling health checks: 2019-11-05 06:26:32,057 INFO scheduling jobs: 2019-11-05 06:26:33,285 DEBUG Dhanu => PING(20) 2019-11-05 06:26:37,036 DEBUG lava-logs => PING(20) 2019-11-05 06:26:52,072 INFO scheduling health checks: 2019-11-05 06:26:52,081 INFO scheduling jobs: 2019-11-05 06:26:53,301 DEBUG Dhanu => PING(20) 2019-11-05 06:26:57,053 DEBUG lava-logs => PING(20) 2019-11-05 06:27:12,095 INFO scheduling health checks: 2019-11-05 06:27:12,106 INFO scheduling jobs: 2019-11-05 06:27:13,319 DEBUG Dhanu => PING(20) 2019-11-05 06:27:17,073 DEBUG lava-logs => PING(20) 2019-11-05 06:27:32,121 INFO scheduling health checks: 2019-11-05 06:27:32,131 INFO scheduling jobs: 2019-11-05 06:27:33,333 DEBUG Dhanu => PING(20) 2019-11-05 06:27:37,093 DEBUG lava-logs => PING(20) 2019-11-05 06:27:52,144 INFO scheduling health checks: 2019-11-05 06:27:52,150 INFO scheduling jobs: 2019-11-05 06:27:53,349 DEBUG Dhanu => PING(20) 2019-11-05 06:27:57,113 DEBUG lava-logs => PING(20) 2019-11-05 06:28:12,164 INFO scheduling health checks: 2019-11-05 06:28:12,177 INFO scheduling jobs: 2019-11-05 06:28:13,373 DEBUG Dhanu => PING(20) 2019-11-05 06:28:17,133 DEBUG lava-logs => PING(20) 2019-11-05 06:28:32,191 INFO scheduling health checks: 2019-11-05 06:28:32,201 INFO scheduling jobs: 2019-11-05 06:28:33,396 DEBUG Dhanu => PING(20) 2019-11-05 06:28:37,156 DEBUG lava-logs => PING(20) 2019-11-05 06:28:52,212 INFO scheduling health checks: 2019-11-05 06:28:52,222 INFO scheduling jobs: 2019-11-05 06:28:53,418 DEBUG Dhanu => PING(20) 2019-11-05 06:28:57,177 DEBUG lava-logs => PING(20) 2019-11-05 06:29:12,237 INFO scheduling health checks: 2019-11-05 06:29:12,247 INFO scheduling jobs: 2019-11-05 06:29:13,438 DEBUG Dhanu => PING(20) 2019-11-05 06:29:17,197 DEBUG lava-logs => PING(20) 2019-11-05 06:29:32,260 INFO scheduling health checks: 2019-11-05 06:29:32,271 INFO scheduling jobs: 2019-11-05 06:29:33,458 DEBUG Dhanu => PING(20) 2019-11-05 06:29:37,217 DEBUG lava-logs => PING(20) 2019-11-05 06:29:52,284 INFO scheduling health checks: 2019-11-05 06:29:52,293 INFO scheduling jobs: 2019-11-05 06:29:53,477 DEBUG Dhanu => PING(20) 2019-11-05 06:29:57,239 DEBUG lava-logs => PING(20) 2019-11-05 06:30:12,308 INFO scheduling health checks: 2019-11-05 06:30:12,318 INFO scheduling jobs: 2019-11-05 06:30:13,493 DEBUG Dhanu => PING(20) 2019-11-05 06:30:17,257 DEBUG lava-logs => PING(20) 2019-11-05 06:30:32,331 INFO scheduling health checks: 2019-11-05 06:30:32,340 INFO scheduling jobs: 2019-11-05 06:30:33,509 DEBUG Dhanu => PING(20) 2019-11-05 06:30:37,277 DEBUG lava-logs => PING(20) 2019-11-05 06:30:52,356 INFO scheduling health checks: 2019-11-05 06:30:52,367 INFO scheduling jobs: 2019-11-05 06:30:53,525 DEBUG Dhanu => PING(20) 2019-11-05 06:30:57,297 DEBUG lava-logs => PING(20) 2019-11-05 06:31:12,380 INFO scheduling health checks: 2019-11-05 06:31:12,392 INFO scheduling jobs: 2019-11-05 06:31:13,544 DEBUG Dhanu => PING(20) 2019-11-05 06:31:17,319 DEBUG lava-logs => PING(20) 2019-11-05 06:31:32,408 INFO scheduling health checks: 2019-11-05 06:31:32,420 INFO scheduling jobs: 2019-11-05 06:31:33,561 DEBUG Dhanu => PING(20) 2019-11-05 06:31:37,337 DEBUG lava-logs => PING(20) 2019-11-05 06:31:52,432 INFO scheduling health checks: 2019-11-05 06:31:52,441 INFO scheduling jobs: 2019-11-05 06:31:53,577 DEBUG Dhanu => PING(20) 2019-11-05 06:31:57,360 DEBUG lava-logs => PING(20) 2019-11-05 06:32:12,455 INFO scheduling health checks: 2019-11-05 06:32:12,466 INFO scheduling jobs: 2019-11-05 06:32:13,593 DEBUG Dhanu => PING(20) 2019-11-05 06:32:17,376 DEBUG lava-logs => PING(20) 2019-11-05 06:32:32,480 INFO scheduling health checks: 2019-11-05 06:32:32,485 INFO scheduling jobs: 2019-11-05 06:32:33,612 DEBUG Dhanu => PING(20) 2019-11-05 06:32:37,393 DEBUG lava-logs => PING(20) 2019-11-05 06:32:52,496 INFO scheduling health checks: 2019-11-05 06:32:52,502 INFO scheduling jobs: 2019-11-05 06:32:53,633 DEBUG Dhanu => PING(20) 2019-11-05 06:32:57,413 DEBUG lava-logs => PING(20) 2019-11-05 06:33:12,514 INFO scheduling health checks: 2019-11-05 06:33:12,520 INFO scheduling jobs: 2019-11-05 06:33:13,649 DEBUG Dhanu => PING(20) 2019-11-05 06:33:17,429 DEBUG lava-logs => PING(20) 2019-11-05 06:33:32,532 INFO scheduling health checks: 2019-11-05 06:33:32,541 INFO scheduling jobs: 2019-11-05 06:33:33,666 DEBUG Dhanu => PING(20) 2019-11-05 06:33:37,445 DEBUG lava-logs => PING(20) 2019-11-05 06:33:52,556 INFO scheduling health checks: 2019-11-05 06:33:52,562 INFO scheduling jobs: 2019-11-05 06:33:53,682 DEBUG Dhanu => PING(20) 2019-11-05 06:33:57,465 DEBUG lava-logs => PING(20) 2019-11-05 06:34:12,574 INFO scheduling health checks: 2019-11-05 06:34:12,581 INFO scheduling jobs: 2019-11-05 06:34:13,697 DEBUG Dhanu => PING(20) 2019-11-05 06:34:17,485 DEBUG lava-logs => PING(20) 2019-11-05 06:34:32,595 INFO scheduling health checks: 2019-11-05 06:34:32,600 INFO scheduling jobs: 2019-11-05 06:34:33,713 DEBUG Dhanu => PING(20) 2019-11-05 06:34:37,500 DEBUG lava-logs => PING(20) 2019-11-05 06:34:52,612 INFO scheduling health checks: 2019-11-05 06:34:52,617 INFO scheduling jobs: 2019-11-05 06:34:53,729 DEBUG Dhanu => PING(20) 2019-11-05 06:34:57,522 DEBUG lava-logs => PING(20) 2019-11-05 06:35:12,628 INFO scheduling health checks: 2019-11-05 06:35:12,633 INFO scheduling jobs: 2019-11-05 06:35:13,745 DEBUG Dhanu => PING(20) 2019-11-05 06:35:17,541 DEBUG lava-logs => PING(20) 2019-11-05 06:35:32,647 INFO scheduling health checks: 2019-11-05 06:35:32,654 INFO scheduling jobs: 2019-11-05 06:35:33,760 DEBUG Dhanu => PING(20) 2019-11-05 06:35:37,557 DEBUG lava-logs => PING(20) 2019-11-05 06:35:52,667 INFO scheduling health checks: 2019-11-05 06:35:52,674 INFO scheduling jobs: 2019-11-05 06:35:53,777 DEBUG Dhanu => PING(20) 2019-11-05 06:35:57,573 DEBUG lava-logs => PING(20) 2019-11-05 06:36:12,688 INFO scheduling health checks: 2019-11-05 06:36:12,694 INFO scheduling jobs: 2019-11-05 06:36:13,797 DEBUG Dhanu => PING(20) 2019-11-05 06:36:17,593 DEBUG lava-logs => PING(20) 2019-11-05 06:36:32,708 INFO scheduling health checks: 2019-11-05 06:36:32,719 INFO scheduling jobs: 2019-11-05 06:36:33,813 DEBUG Dhanu => PING(20) 2019-11-05 06:36:37,609 DEBUG lava-logs => PING(20) 2019-11-05 06:36:52,728 INFO scheduling health checks: 2019-11-05 06:36:52,735 INFO scheduling jobs: 2019-11-05 06:36:53,829 DEBUG Dhanu => PING(20) 2019-11-05 06:36:57,626 DEBUG lava-logs => PING(20) 2019-11-05 06:37:12,748 INFO scheduling health checks: 2019-11-05 06:37:12,755 INFO scheduling jobs: 2019-11-05 06:37:13,845 DEBUG Dhanu => PING(20) 2019-11-05 06:37:17,645 DEBUG lava-logs => PING(20) 2019-11-05 06:37:32,768 INFO scheduling health checks: 2019-11-05 06:37:32,773 INFO scheduling jobs: 2019-11-05 06:37:33,866 DEBUG Dhanu => PING(20) 2019-11-05 06:37:37,660 DEBUG lava-logs => PING(20) 2019-11-05 06:37:52,784 INFO scheduling health checks: 2019-11-05 06:37:52,792 INFO scheduling jobs: 2019-11-05 06:37:53,880 DEBUG Dhanu => PING(20) 2019-11-05 06:37:57,674 DEBUG lava-logs => PING(20) 2019-11-05 06:38:12,804 INFO scheduling health checks: 2019-11-05 06:38:12,809 INFO scheduling jobs: 2019-11-05 06:38:13,902 DEBUG Dhanu => PING(20) 2019-11-05 06:38:17,693 DEBUG lava-logs => PING(20) 2019-11-05 06:38:32,827 INFO scheduling health checks: 2019-11-05 06:38:32,832 INFO scheduling jobs: 2019-11-05 06:38:33,917 DEBUG Dhanu => PING(20) 2019-11-05 06:38:37,709 DEBUG lava-logs => PING(20) 2019-11-05 06:38:52,844 INFO scheduling health checks: 2019-11-05 06:38:52,849 INFO scheduling jobs: 2019-11-05 06:38:53,937 DEBUG Dhanu => PING(20) 2019-11-05 06:38:57,725 DEBUG lava-logs => PING(20) 2019-11-05 06:39:12,864 INFO scheduling health checks: 2019-11-05 06:39:12,869 INFO scheduling jobs: 2019-11-05 06:39:13,956 DEBUG Dhanu => PING(20) 2019-11-05 06:39:17,743 DEBUG lava-logs => PING(20) 2019-11-05 06:39:32,880 INFO scheduling health checks: 2019-11-05 06:39:32,888 INFO scheduling jobs: 2019-11-05 06:39:33,973 DEBUG Dhanu => PING(20) 2019-11-05 06:39:37,764 DEBUG lava-logs => PING(20) 2019-11-05 06:39:52,899 INFO scheduling health checks: 2019-11-05 06:39:52,903 INFO scheduling jobs: 2019-11-05 06:39:53,989 DEBUG Dhanu => PING(20) 2019-11-05 06:39:57,785 DEBUG lava-logs => PING(20) 2019-11-05 06:40:12,912 INFO scheduling health checks: 2019-11-05 06:40:12,918 INFO scheduling jobs: 2019-11-05 06:40:14,006 DEBUG Dhanu => PING(20) 2019-11-05 06:40:17,803 DEBUG lava-logs => PING(20) 2019-11-05 06:40:32,932 INFO scheduling health checks: 2019-11-05 06:40:32,939 INFO scheduling jobs: 2019-11-05 06:40:34,025 DEBUG Dhanu => PING(20) 2019-11-05 06:40:37,820 DEBUG lava-logs => PING(20) 2019-11-05 06:40:52,952 INFO scheduling health checks: 2019-11-05 06:40:52,959 INFO scheduling jobs: 2019-11-05 06:40:54,041 DEBUG Dhanu => PING(20) 2019-11-05 06:40:57,837 DEBUG lava-logs => PING(20) 2019-11-05 06:41:12,972 INFO scheduling health checks: 2019-11-05 06:41:12,980 INFO scheduling jobs: 2019-11-05 06:41:14,057 DEBUG Dhanu => PING(20) 2019-11-05 06:41:17,859 DEBUG lava-logs => PING(20) 2019-11-05 06:41:32,992 INFO scheduling health checks: 2019-11-05 06:41:32,996 INFO scheduling jobs: 2019-11-05 06:41:34,073 DEBUG Dhanu => PING(20) 2019-11-05 06:41:37,875 DEBUG lava-logs => PING(20) 2019-11-05 06:41:53,007 INFO scheduling health checks: 2019-11-05 06:41:53,016 INFO scheduling jobs: 2019-11-05 06:41:54,089 DEBUG Dhanu => PING(20) 2019-11-05 06:41:57,893 DEBUG lava-logs => PING(20) 2019-11-05 06:42:13,028 INFO scheduling health checks: 2019-11-05 06:42:13,035 INFO scheduling jobs: 2019-11-05 06:42:14,106 DEBUG Dhanu => PING(20) 2019-11-05 06:42:17,908 DEBUG lava-logs => PING(20) 2019-11-05 06:42:33,048 INFO scheduling health checks: 2019-11-05 06:42:33,054 INFO scheduling jobs: 2019-11-05 06:42:34,121 DEBUG Dhanu => PING(20) 2019-11-05 06:42:37,927 DEBUG lava-logs => PING(20) 2019-11-05 06:42:53,067 INFO scheduling health checks: 2019-11-05 06:42:53,073 INFO scheduling jobs: 2019-11-05 06:42:54,136 DEBUG Dhanu => PING(20) 2019-11-05 06:42:57,945 DEBUG lava-logs => PING(20) 2019-11-05 06:43:13,085 INFO scheduling health checks: 2019-11-05 06:43:13,090 INFO scheduling jobs: 2019-11-05 06:43:14,154 DEBUG Dhanu => PING(20) 2019-11-05 06:43:17,965 DEBUG lava-logs => PING(20) 2019-11-05 06:43:33,100 INFO scheduling health checks: 2019-11-05 06:43:33,108 INFO scheduling jobs: 2019-11-05 06:43:34,173 DEBUG Dhanu => PING(20) 2019-11-05 06:43:37,983 DEBUG lava-logs => PING(20) 2019-11-05 06:43:53,120 INFO scheduling health checks: 2019-11-05 06:43:53,126 INFO scheduling jobs: 2019-11-05 06:43:54,185 DEBUG Dhanu => PING(20) 2019-11-05 06:43:58,001 DEBUG lava-logs => PING(20) 2019-11-05 06:44:13,136 INFO scheduling health checks: 2019-11-05 06:44:13,145 INFO scheduling jobs: 2019-11-05 06:44:14,201 DEBUG Dhanu => PING(20) 2019-11-05 06:44:18,021 DEBUG lava-logs => PING(20) 2019-11-05 06:44:33,156 INFO scheduling health checks: 2019-11-05 06:44:33,162 INFO scheduling jobs: 2019-11-05 06:44:34,217 DEBUG Dhanu => PING(20) 2019-11-05 06:44:38,037 DEBUG lava-logs => PING(20) 2019-11-05 06:44:53,176 INFO scheduling health checks: 2019-11-05 06:44:53,182 INFO scheduling jobs: 2019-11-05 06:44:54,234 DEBUG Dhanu => PING(20) 2019-11-05 06:44:58,052 DEBUG lava-logs => PING(20) 2019-11-05 06:45:13,196 INFO scheduling health checks: 2019-11-05 06:45:13,202 INFO scheduling jobs: 2019-11-05 06:45:14,254 DEBUG Dhanu => PING(20) 2019-11-05 06:45:18,071 DEBUG lava-logs => PING(20) 2019-11-05 06:45:33,212 INFO scheduling health checks: 2019-11-05 06:45:33,219 INFO scheduling jobs: 2019-11-05 06:45:34,274 DEBUG Dhanu => PING(20) 2019-11-05 06:45:38,091 DEBUG lava-logs => PING(20) 2019-11-05 06:45:53,232 INFO scheduling health checks: 2019-11-05 06:45:53,238 INFO scheduling jobs: 2019-11-05 06:45:54,292 DEBUG Dhanu => PING(20) 2019-11-05 06:45:58,109 DEBUG lava-logs => PING(20) 2019-11-05 06:46:13,248 INFO scheduling health checks: 2019-11-05 06:46:13,256 INFO scheduling jobs: 2019-11-05 06:46:14,309 DEBUG Dhanu => PING(20) 2019-11-05 06:46:18,125 DEBUG lava-logs => PING(20) 2019-11-05 06:46:33,268 INFO scheduling health checks: 2019-11-05 06:46:33,275 INFO scheduling jobs: 2019-11-05 06:46:34,323 DEBUG Dhanu => PING(20) 2019-11-05 06:46:38,145 DEBUG lava-logs => PING(20) 2019-11-05 06:46:53,288 INFO scheduling health checks: 2019-11-05 06:46:53,295 INFO scheduling jobs: 2019-11-05 06:46:54,341 DEBUG Dhanu => PING(20) 2019-11-05 06:46:58,165 DEBUG lava-logs => PING(20) 2019-11-05 06:47:13,308 INFO scheduling health checks: 2019-11-05 06:47:13,315 INFO scheduling jobs: 2019-11-05 06:47:14,356 DEBUG Dhanu => PING(20) 2019-11-05 06:47:18,185 DEBUG lava-logs => PING(20) 2019-11-05 06:47:33,328 INFO scheduling health checks: 2019-11-05 06:47:33,332 INFO scheduling jobs: 2019-11-05 06:47:34,372 DEBUG Dhanu => PING(20) 2019-11-05 06:47:38,200 DEBUG lava-logs => PING(20) 2019-11-05 06:47:53,343 INFO scheduling health checks: 2019-11-05 06:47:53,347 INFO scheduling jobs: 2019-11-05 06:47:54,389 DEBUG Dhanu => PING(20) 2019-11-05 06:47:58,217 DEBUG lava-logs => PING(20) 2019-11-05 06:48:13,360 INFO scheduling health checks: 2019-11-05 06:48:13,364 INFO scheduling jobs: 2019-11-05 06:48:14,405 DEBUG Dhanu => PING(20) 2019-11-05 06:48:18,233 DEBUG lava-logs => PING(20) 2019-11-05 06:48:33,375 INFO scheduling health checks: 2019-11-05 06:48:33,383 INFO scheduling jobs: 2019-11-05 06:48:34,420 DEBUG Dhanu => PING(20) 2019-11-05 06:48:38,252 DEBUG lava-logs => PING(20) 2019-11-05 06:48:53,399 INFO scheduling health checks: 2019-11-05 06:48:53,405 INFO scheduling jobs: 2019-11-05 06:48:54,437 DEBUG Dhanu => PING(20) 2019-11-05 06:48:58,269 DEBUG lava-logs => PING(20) 2019-11-05 06:49:13,417 INFO scheduling health checks: 2019-11-05 06:49:13,428 INFO scheduling jobs: 2019-11-05 06:49:14,453 DEBUG Dhanu => PING(20) 2019-11-05 06:49:18,287 DEBUG lava-logs => PING(20) 2019-11-05 06:49:33,444 INFO scheduling health checks: 2019-11-05 06:49:33,449 INFO scheduling jobs: 2019-11-05 06:49:34,469 DEBUG Dhanu => PING(20) 2019-11-05 06:49:38,309 DEBUG lava-logs => PING(20) 2019-11-05 06:49:53,464 INFO scheduling health checks: 2019-11-05 06:49:53,475 INFO scheduling jobs: 2019-11-05 06:49:54,484 DEBUG Dhanu => PING(20) 2019-11-05 06:49:58,325 DEBUG lava-logs => PING(20) 2019-11-05 06:50:13,488 INFO scheduling health checks: 2019-11-05 06:50:13,497 INFO scheduling jobs: 2019-11-05 06:50:14,501 DEBUG Dhanu => PING(20) 2019-11-05 06:50:18,341 DEBUG lava-logs => PING(20) 2019-11-05 06:50:33,508 INFO scheduling health checks: 2019-11-05 06:50:33,516 INFO scheduling jobs: 2019-11-05 06:50:34,517 DEBUG Dhanu => PING(20) 2019-11-05 06:50:38,357 DEBUG lava-logs => PING(20) 2019-11-05 06:50:53,532 INFO scheduling health checks: 2019-11-05 06:50:53,536 INFO scheduling jobs: 2019-11-05 06:50:54,533 DEBUG Dhanu => PING(20) 2019-11-05 06:50:58,379 DEBUG lava-logs => PING(20) 2019-11-05 06:51:13,549 INFO scheduling health checks: 2019-11-05 06:51:13,554 INFO scheduling jobs: 2019-11-05 06:51:14,549 DEBUG Dhanu => PING(20) 2019-11-05 06:51:18,399 DEBUG lava-logs => PING(20) 2019-11-05 06:51:33,567 INFO scheduling health checks: 2019-11-05 06:51:33,573 INFO scheduling jobs: 2019-11-05 06:51:34,566 DEBUG Dhanu => PING(20) 2019-11-05 06:51:38,417 DEBUG lava-logs => PING(20) 2019-11-05 06:51:53,588 INFO scheduling health checks: 2019-11-05 06:51:53,595 INFO scheduling jobs: 2019-11-05 06:51:54,585 DEBUG Dhanu => PING(20) 2019-11-05 06:51:58,433 DEBUG lava-logs => PING(20) 2019-11-05 06:52:13,613 INFO scheduling health checks: 2019-11-05 06:52:13,621 INFO scheduling jobs: 2019-11-05 06:52:14,600 DEBUG Dhanu => PING(20) 2019-11-05 06:52:18,449 DEBUG lava-logs => PING(20) 2019-11-05 06:52:33,632 INFO scheduling health checks: 2019-11-05 06:52:33,639 INFO scheduling jobs: 2019-11-05 06:52:34,617 DEBUG Dhanu => PING(20) 2019-11-05 06:52:38,469 DEBUG lava-logs => PING(20) 2019-11-05 06:52:53,652 INFO scheduling health checks: 2019-11-05 06:52:53,661 INFO scheduling jobs: 2019-11-05 06:52:54,633 DEBUG Dhanu => PING(20) 2019-11-05 06:52:58,487 DEBUG lava-logs => PING(20) 2019-11-05 06:53:13,680 INFO scheduling health checks: 2019-11-05 06:53:13,691 INFO scheduling jobs: 2019-11-05 06:53:14,657 DEBUG Dhanu => PING(20) 2019-11-05 06:53:18,510 DEBUG lava-logs => PING(20) 2019-11-05 06:53:33,705 INFO scheduling health checks: 2019-11-05 06:53:33,713 INFO scheduling jobs: 2019-11-05 06:53:34,673 DEBUG Dhanu => PING(20) 2019-11-05 06:53:38,529 DEBUG lava-logs => PING(20) 2019-11-05 06:53:53,724 INFO scheduling health checks: 2019-11-05 06:53:53,733 INFO scheduling jobs: 2019-11-05 06:53:54,689 DEBUG Dhanu => PING(20) 2019-11-05 06:53:58,548 DEBUG lava-logs => PING(20) 2019-11-05 06:54:13,748 INFO scheduling health checks: 2019-11-05 06:54:13,753 INFO scheduling jobs: 2019-11-05 06:54:14,711 DEBUG Dhanu => PING(20) 2019-11-05 06:54:18,569 DEBUG lava-logs => PING(20) 2019-11-05 06:54:33,765 INFO scheduling health checks: 2019-11-05 06:54:33,775 INFO scheduling jobs: 2019-11-05 06:54:34,729 DEBUG Dhanu => PING(20) 2019-11-05 06:54:38,589 DEBUG lava-logs => PING(20) 2019-11-05 06:54:53,788 INFO scheduling health checks: 2019-11-05 06:54:53,800 INFO scheduling jobs: 2019-11-05 06:54:54,749 DEBUG Dhanu => PING(20) 2019-11-05 06:54:58,609 DEBUG lava-logs => PING(20) 2019-11-05 06:55:13,812 INFO scheduling health checks: 2019-11-05 06:55:13,823 INFO scheduling jobs: 2019-11-05 06:55:14,771 DEBUG Dhanu => PING(20) 2019-11-05 06:55:18,632 DEBUG lava-logs => PING(20) 2019-11-05 06:55:33,836 INFO scheduling health checks: 2019-11-05 06:55:33,843 INFO scheduling jobs: 2019-11-05 06:55:34,795 DEBUG Dhanu => PING(20) 2019-11-05 06:55:38,654 DEBUG lava-logs => PING(20) 2019-11-05 06:55:53,856 INFO scheduling health checks: 2019-11-05 06:55:53,865 INFO scheduling jobs: 2019-11-05 06:55:54,817 DEBUG Dhanu => PING(20) 2019-11-05 06:55:58,677 DEBUG lava-logs => PING(20) 2019-11-05 06:56:13,880 INFO scheduling health checks: 2019-11-05 06:56:13,895 INFO scheduling jobs: 2019-11-05 06:56:14,837 DEBUG Dhanu => PING(20) 2019-11-05 06:56:18,700 DEBUG lava-logs => PING(20) 2019-11-05 06:56:33,916 INFO scheduling health checks: 2019-11-05 06:56:33,926 INFO scheduling jobs: 2019-11-05 06:56:34,853 DEBUG Dhanu => PING(20) 2019-11-05 06:56:38,713 DEBUG lava-logs => PING(20) 2019-11-05 06:56:53,940 INFO scheduling health checks: 2019-11-05 06:56:53,951 INFO scheduling jobs: 2019-11-05 06:56:54,880 DEBUG Dhanu => PING(20) 2019-11-05 06:56:58,736 DEBUG lava-logs => PING(20) 2019-11-05 06:57:13,968 INFO scheduling health checks: 2019-11-05 06:57:13,979 INFO scheduling jobs: 2019-11-05 06:57:14,901 DEBUG Dhanu => PING(20) 2019-11-05 06:57:18,750 DEBUG lava-logs => PING(20) 2019-11-05 06:57:33,992 INFO scheduling health checks: 2019-11-05 06:57:34,004 INFO scheduling jobs: 2019-11-05 06:57:34,917 DEBUG Dhanu => PING(20) 2019-11-05 06:57:38,773 DEBUG lava-logs => PING(20) 2019-11-05 06:57:54,016 INFO scheduling health checks: 2019-11-05 06:57:54,027 INFO scheduling jobs: 2019-11-05 06:57:54,938 DEBUG Dhanu => PING(20) 2019-11-05 06:57:58,794 DEBUG lava-logs => PING(20) 2019-11-05 06:58:14,041 INFO scheduling health checks: 2019-11-05 06:58:14,052 INFO scheduling jobs: 2019-11-05 06:58:14,958 DEBUG Dhanu => PING(20) 2019-11-05 06:58:18,813 DEBUG lava-logs => PING(20) 2019-11-05 06:58:34,066 INFO scheduling health checks: 2019-11-05 06:58:34,074 INFO scheduling jobs: 2019-11-05 06:58:34,977 DEBUG Dhanu => PING(20) 2019-11-05 06:58:38,833 DEBUG lava-logs => PING(20) 2019-11-05 06:58:54,085 INFO scheduling health checks: 2019-11-05 06:58:54,091 INFO scheduling jobs: 2019-11-05 06:58:54,997 DEBUG Dhanu => PING(20) 2019-11-05 06:58:58,850 DEBUG lava-logs => PING(20) 2019-11-05 06:59:14,104 INFO scheduling health checks: 2019-11-05 06:59:14,110 INFO scheduling jobs: 2019-11-05 06:59:15,014 DEBUG Dhanu => PING(20) 2019-11-05 06:59:18,869 DEBUG lava-logs => PING(20) 2019-11-05 06:59:34,120 INFO scheduling health checks: 2019-11-05 06:59:34,125 INFO scheduling jobs: 2019-11-05 06:59:35,029 DEBUG Dhanu => PING(20) 2019-11-05 06:59:38,890 DEBUG lava-logs => PING(20)