Webroar WebROaR - Ruby Application Server

Comments and changes to this ticket

• 

  Aditya Babbar December 7th, 2009 @ 03:06 PM

  • Assigned user set to “Aditya Babbar”
  • State changed from “new” to “open”

  Could you please provide us the error messages in the log files on your system - 'webroar.log' and 'Admin Panel.log' available in /var/log/webroar? These would help in further debugging of the issue.

  Thanks.

• You flagged this item as spam.
  

  eveel December 7th, 2009 @ 08:06 PM

  Certainly, but I have also one small comment, maybe it is important:

  # /sbin/service webroar start
  Initiating WebROaR startup sequence ...
  Starting message queue server ... failed.
  'Analytics' and 'Exception Notification' features would not work. Please refer '/var/log/webroar/analyzer.log' for details.
  Starting webroar-head process ...
  No applications currently deployed on the server.
  Head process started successfully.
  Server started but 'Analytics' and 'Exception Notification' features would not work.
  

  access.log fragment:

  192.168.0.104 - - [07/Dec/2009:19:34:59 +0500] "GET /admin-panel HTTP/1.1" 404 259 "-" "Opera/9.80 (X11; Linux i686; U; ru) Presto/2.2.15 Version/10.10"
  192.168.0.104 - - [07/Dec/2009:19:34:59 +0500] "GET /favicon.ico HTTP/1.1" 404 259 "http://192.168.0.106:3000/admin-panel" "Opera/9.80 (X11; Linux i686; U; ru) Presto/2.2.15 Version/10.10"
  192.168.0.104 - - [07/Dec/2009:19:35:07 +0500] "GET / HTTP/1.1" 404 259 "-" "Opera/9.80 (X11; Linux i686; U; ru) Presto/2.2.15 Version/10.10"
  

  webroar.log fragment:

  Log file opened at Mon Dec  7 19:34:32 2009
  Mon Dec  7 19:34:32 2009-6581-Info:No applications currently deployed on the server.
  Mon Dec  7 19:34:32 2009-6581-Info:No applications currently deployed on the server.
  Mon Dec  7 19:34:32 2009-6581-Info:setting log level to SEVERE
  Mon Dec  7 19:34:32 2009-6581-Info:Network server successfully initialized on port 3000
  Mon Dec  7 19:34:32 2009-6581-Info:Controller initialized
  Mon Dec  7 19:34:32 2009-6581-Info:PID of created worker = 6582, Rails application=static-worker, 
  Mon Dec  7 19:34:32 2009-6581-Info:PID of created worker = 6583, Rails application=static-worker, 
  Mon Dec  7 19:34:32 2009-6581-Info:PID of created worker = 6584, Rails application=static-worker, 
  Mon Dec  7 19:34:32 2009-6581-Info:PID of created worker = 6585, Rails application=static-worker, 
  Mon Dec  7 19:34:32 2009-6581-Info:PID of created worker = 6586, Rails application=/opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/webroar-0.2.4/src/admin_panel, 
  Mon Dec  7 19:34:57 2009-6587-Info:wr_app_wrk_add_timeout_cb: killing worker, pid = 6586
  Mon Dec  7 19:34:57 2009-6587-Info:wr_app_wrk_add_timeout_cb: killing worker, pid = 6582
  

  Also I ahve no file named Admin Panel.log in /var/log/webroar directory.

• You flagged this item as spam.
  

  eveel December 7th, 2009 @ 08:10 PM

  Also, server shutdown:

  # /sbin/service webroar stop
  Stopping webroar-head ... done.
  Stopping webroar-analyzer process ... failed.
  Failed to retrieve pid for webroar-analyzer process. Unable to kill it.
  

  My first anamnesis was "webroar can't find some (starling) binaries in non-standard PATH). After I changed file /etc/init.d/webroar and added line export PATH=/opt/ruby/bin:$PATH, WebROaR startup looks like:

  # /sbin/service webroar start
  Initiating WebROaR startup sequence ...
  Starting message queue server ... done.
  Starting webroar-head process ...
  No applications currently deployed on the server.
  Head process started successfully.
  Starting webroar-analyzer process ... done.
  Server started successfully.
  

  But there's no changes for me.

• 

  Aditya Babbar December 8th, 2009 @ 05:14 PM

  Thank you for providing all the details and making all these additional efforts. Yes, WebROaR does start up Starling Message Queue server to which all worker processes post analytics & exceptions data packets. Analyzer process dequeues them and dumps them to a database after processing & massaging the data.

  The core issue being seen on your setup is a bit different. Head process forks & execl's the required number of worker processes, and waits for them to connect back to it (over a Unix Domain Socket). From the logs, it looks like none of the workers are able to connect back. Either execl() didn't work as expected or they crashed without even being able to create their individual log files. ('Admin Panel.log' and 'static-worker.log')

  • Could you please try installing the debug build of WebROaR that would generate more verbose logs and help us in pin pointing the issue? (execl() related log messages would appear in webroar.log). You can use the following commands:

      sudo webroar uninstall
      sudo webroar install --debug-build
  

  • This is just a shot in the dark. Is there any SELinux policy that might interfere with this fork/execl/connect calls being made by the server?

• You flagged this item as spam.
  

  eveel December 9th, 2009 @ 04:51 AM

  No, SELinux is disabled:
  

  root{/var/log/webroar}# /usr/sbin/selinuxenabled; echo $?
  1
  

  Okay, I've compiled and installed the debug build of WebROaR. Check the logs in attachment or here: http://pastie.org/734512 .

  • webroar.log 15.8 KB
• 

  Aditya Babbar December 15th, 2009 @ 12:38 PM

  • State changed from “open” to “resolved”

  Thanks a lot for all your efforts.

  We have nailed down the issue and the latest code in the Github repository has the fix for it. Could you please try it out and let us know how it goes?

  Essentially, there was a conflict in the name of a method 'log' defined by us using rb_define_singleton_method(). It was colliding with the one required by tcmalloc library used by REE. The following trace let us help in tracking this issue down. (In fact we anyway were renaming this method to avoid another collision with the method 'log' used by Math.log. Refer Ticket #10 for more details.)

  Program received signal SIGSEGV, Segmentation fault.
  st_lookup (table=0x0, key=140737336659901, value=0x7fffffffe150) at st.c:250
  250        hash_val = do_hash(key, table);
  (gdb) bt
  #0  st_lookup (table=0x0, key=140737336659901, value=0x7fffffffe150) at st.c:250
  #1  0x00007ffff6f23241 in rb_intern (name=0x7ffff6f54fbd "to_str") at
  parse.y:6163
  #2  0x00007ffff6f1e7d1 in convert_type (val=4097, tname=0x7ffff6f54fbd "to_str", method=0x7fffffffe150 "", raise=2) at object.c:2207
  #3  0x00007ffff6f1ea61 in rb_convert_type (val=4097, type=7, tname=0x7ffff6f6b4fd "String", method=0x7ffff6f54fbd "to_str")
      at object.c:2233
  #4  0x00007ffff6f47e14 in rb_string_value (ptr=0x7fffffffe230) at
  string.c:566
  #5  0x00007ffff6f4a344 in rb_string_value_ptr (ptr=0x0) at string.c:579
  #6  0x000000000040aba8 in log (_=<value optimized out>, message_type=140737336659901, severity=140737488347472, log_message=2)
      at
  /opt/ree/lib/ruby/gems/1.8/gems/webroar-0.2.4/src/worker/wkr_http.c:151
  #7  0x00007ffff6a6ebf4 in tcmalloc::Sampler::PopulateFastLog2Table () at
  src/sampler.cc:68
  #8  0x00007ffff6a6f438 in tcmalloc::Static::InitStaticVars () at
  src/static_vars.cc:62
  #9  0x00007ffff6a70545 in tcmalloc::ThreadCache::InitModule () at src/thread_cache.cc:310 #10 0x00007ffff6a75765 in tcmalloc::ThreadCache::GetCache (size=30) at
  src/thread_cache.h:378
  #11 do_malloc (size=30) at src/tcmalloc.cc:794
  #12 cpp_alloc (size=30) at src/tcmalloc.cc:1069
  #13 tc_new (size=30) at src/tcmalloc.cc:1185
  #14 0x00007ffff67f3cb1 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) ()
     from /usr/lib/libstdc++.so.6
  #15 0x00007ffff67f48e5 in ?? () from /usr/lib/libstdc++.so.6
  #16 0x00007ffff67f4a82 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib/libstdc++.so.6
  #17 0x00007ffff6a6f8e1 in __static_initialization_and_destruction_0 () at src/symbolize.cc:59
  #18 global constructors keyed to symbolize.cc () at src/symbolize.cc:162
  #19 0x00007ffff6a730d6 in __do_global_ctors_aux () from /opt/ree/lib/libtcmalloc_minimal.so.0
  #20 0x00007ffff6a6840b in _init () from /opt/ree/lib/libtcmalloc_minimal.so.0
  

• You flagged this item as spam.
  

  eveel December 15th, 2009 @ 03:56 PM

  Ah, yeah! Now WebROaR works fine and we'll try it in production.

  Thanks.