A partial archive of meta.discourse.org as of Tuesday July 18, 2017.

Tons of Redis Errors

jamesmarkcook

I've recently been seeing this in my log files.

Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 
Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
4 ActionView::Template::Error (MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk.
Commands that may modify the data set are disabled. Please check Redis logs for details about the error.)
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:199:in `call_pipelined' web-exception

Has anybody come across this recently?

jamesmarkcook

This just happened again:

Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 
Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 
2 Failed to process job: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 
Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Stacktrace:

"/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:199:in `call_pipelined'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:157:in `block in call_pipeline'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:293:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:155:in `call_pipeline'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2299:in `block in multi'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:58:in `block in synchronize'", "/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis.rb:2291:in `multi'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus/backends/redis.rb:91:in `publish'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus.rb:233:in `publish'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus.rb:468:in `block in new_subscriber_thread'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus/timer_thread.rb:98:in `do_work'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.1/lib/message_bus/timer_thread.rb:29:in `block in initialize'"]

Apologies, can't format that correctly. This seems to happen when I'm using the console.

Falco

Disk Space is plenty?

jamesmarkcook

Should be. 2GB Digital Ocean droplet created 21 days ago. Forum not being used yet.

Falco

Can you run a df -h ?

jamesmarkcook
Filesystem                 Size  Used Avail Use% Mounted on
udev                       991M  4.0K  991M   1% /dev
tmpfs                      201M  348K  200M   1% /run
/dev/disk/by-label/DOROOT   40G   11G   27G  29% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                      1001M  588K 1001M   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        40G   11G   27G  29% /var/lib/docker/aufs/mnt/e01e39c23091db7cede7da8673b5a4efc30eaa5252247ecbe472e40aa28ccbdd
shm                         64M  4.0K   64M   1% /var/lib/docker/containers/5453b058278d67388a26b65f26daf0737b4c05e20a0add38a83d9282cb493246/shm

Thanks

mpalmer

You probably want to start here. No point going off on wild goose chases if the logs will tell you exactly what went wrong.

jamesmarkcook

Excuse my ignorance. Where can I find these logs? I can't see them in the log folder. Thanks

jamesmarkcook

It doesn't look like a logfile is set for Redis. Is it being caught somewhere else? I can't see much when running ./launcher logs app

mpalmer

Redis should be logging to the container's stdout, so the log entries should be in the output of ./launcher logs app. If they're not, you've got a whole other level of o.O going on.

jamesmarkcook

Thanks. I will keep an eye on these logs and report anything of note. The error only seems to have happened when I've been interacting with the container so far. Either in the console or running the importer script

mpalmer

That smells like lack of memory. Does your host's dmesg (or kernel log) report any oomkiller activity?

jamesmarkcook

You may be correct

dmesg | egrep -i “killed process”

returns

[785023.865810] Killed process 28218 (ruby) total-vm:1841856kB, anon-rss:398216kB, file-rss:0kB
[808096.474228] Killed process 26908 (ruby) total-vm:1804880kB, anon-rss:386624kB, file-rss:0kB
[808589.865843] Killed process 24691 (ruby) total-vm:1814288kB, anon-rss:374276kB, file-rss:0kB
[808589.874840] Killed process 24775 (100-sidekiq.rb*) total-vm:1814288kB, anon-rss:375152kB, file-rss:0kB
[808637.769527] Killed process 3212 (ruby) total-vm:1755040kB, anon-rss:290624kB, file-rss:0kB
[811306.837021] Killed process 12728 (convert) total-vm:205300kB, anon-rss:127676kB, file-rss:0kB
[811730.833113] Killed process 19230 (ruby) total-vm:1851664kB, anon-rss:362160kB, file-rss:0kB
[811730.837619] Killed process 19232 (ruby-timer-thr) total-vm:1851664kB, anon-rss:363728kB, file-rss:0kB
[811730.912056] Killed process 19397 (ruby) total-vm:1724988kB, anon-rss:290648kB, file-rss:0kB







I have no idea why I'd be running out of memory though. I'm running a fresh install from a DO image. 2GB of memory.

mpalmer

Hmm, I'd expect to see some redis-server OOMs in there, too, because that's usually what kills RDB saves. Have you setup a swap file in your droplet? That helps the system keep going in the face of memory pressure during imports, rebuilds, and so on.

jamesmarkcook

Not in this one yet no. I'll make it a priority and see if that helps. Thanks for your help :thumbsup:

Sung_Kim

Did you figure this out?

Sung_Kim

I also got many errors:
Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk.
Commands that may modify the data set are disabled.

./launcher logs app shows me

41:M 24 Jan 13:29:24.012 # Can't save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:30.024 * 1 changes in 900 seconds. Saving...
41:M 24 Jan 13:29:30.024 # Can't save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:36.042 * 1 changes in 900 seconds. Saving...
41:M 24 Jan 13:29:36.043 # Can't save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:42.054 * 1 changes in 900 seconds. Saving...
41:M 24 Jan 13:29:42.054 # Can't save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:48.067 * 1 changes in 900 seconds. Saving...
41:M 24 Jan 13:29:48.067 # Can't save in background: fork: Cannot allocate memory
41:M 24 Jan 13:29:54.082 * 1 changes in 900 seconds. Saving...
41:M 24 Jan 13:29:54.082 # Can't save in background: fork: Cannot allocate memory









Is this a temporary? I need to restart the server to recover this error.

Any thoughts?

Falco

What's the output of

free -m
bufordtaylor

I’m also experiencing this issue and I’m not sure where to go from here

tail unicorn.stderr.log
Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Mem

free
             total       used       free     shared    buffers     cached
Mem:       2049940    1791900     258040     210160       9532     274436
-/+ buffers/cache:    1507932     542008
Swap:            0          0          0


df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                       991M  4.0K  991M   1% /dev
tmpfs                      201M  356K  200M   1% /run
/dev/disk/by-label/DOROOT   40G   15G   24G  38% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                      1001M  1.2M 1000M   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        40G   15G   24G  38% /var/lib/docker/aufs/mnt/b34e2e05c05cfc59ac267c11137745713e4d779c22dd2da8074e9d840db147f
shm                         64M  4.0K   64M   1% /var/lib/docker/containers/1e10c57a8397d72f999a9dfb5169194558110c6e24adb557229ac4b39b5ad5a/shm

Every day or so, discourse will fail and start serving 500 errors. I inherited this Discourse server and am new to it, so I’m open to reading material to point me in the right direction.

People tend to say “make a swap file.” Am I overthinking this or should I just make a swap file?

codinghorror

It never hurts to have a swap file and it is recommended so I would start with that. Then rebuild from the command line.