Attendees
| IRC nick
|
Real name
|
| apmon |
Kai Krueger
|
| iandees |
Ian Dees
|
| pnorman |
Paul Norman
|
| TomH |
Tom Hughes
|
| zere |
Matt Amos
|
Summary
- Long lines issue:
- apmon will likely tag the current osm2pgsql as stable in a week or so, and then merge zere's patch in afterwards.
- Notes branch:
- Should the delete API call (just a soft delete) be restricted to moderators?
- Since the purpose is to restrict to deletion of spam / offensive stuff, then the consensus seemed to be that restriction to mods / admins was the best behaviour.
- Carto style port:
- ACTION apmon to lookat improving render_speedtest.
- ACTION pnorman to try benchmarking on benzene with existing tools.
IRC Log
17:02:57 <zere> minutes of the last meeting, for your browsing pleasure http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2013-04-01
17:03:12 <TomH> evening
17:03:12 <zere> as for actions in the last meeting:
17:04:38 <apmon> Hi
17:04:52 <zere> i managed to reproduce the issue with long ways, and sent a patch to apmon which cuts up long ways between nodes. this should solve the problem that osm2pgsql has, and i heard some talk over on #osm-dev that other problems in other bits of the stack - e.g: mapnik have also been looked at.
17:04:59 <zere> #topic actions from the last meeting
17:05:18 <zere> anyone know any more about changes to the queries / mapnik that might have gone on?
17:05:21 <pnorman> I've got absolutely nothing done on summarizing the issue for springmeyer
17:05:24 <TomH> I have one q re notes
17:05:46 <apmon> zere: I will likely tag the current osm2pgsql as stable in a week or so, and then merge your patch in afterwards
17:05:52 <TomH> ah yes I missed last week because I was so busy with chef I forgot about it
17:06:08 <zere> apmon: awesome :-)
17:06:58 <zere> the other action was to add something to capabilities about api up/down-ness, and that's at: https://github.com/openstreetmap/openstreetmap-website/commit/3822e8de
17:07:20 <TomH> so notes has a delete api call (just a soft delete) that isn't exposed in the web interface and I'm wondering if it should be restricted to mods? it does now require auth, like close does
17:07:58 <pnorman> if it leaves no publically accessible trace of the note then it should be restricted
17:08:01 <zere> #topic notes branch
17:08:39 <zere> yeah, i think it's probably best - if the purpose is getting rid of spam / offensive reports then it's ok to restrict to mods
17:08:42 <apmon> Yes, the delete call should be handled similarly to the delete call of diary entries, which is restricted to moderator?
17:08:56 <TomH> to admin actually
17:09:39 <apmon> And yes, the purpose of that call was to deal with spam / offensive reports
17:11:33 <TomH> right, so restrict to mod and probably add to web interface for mobs
17:12:13 <apmon> sounds good
17:12:23 <pnorman> how close are we on that? anything we can do to help?
17:12:31 <pnorman> (that=notes branch)
17:12:56 <TomH> this is the only outstanding issue I think
17:14:02 <apmon> :-)
17:16:45 <zere> great. is there anything else anyone wanted to discuss?
17:16:48 <pnorman> couple of points on carto port
17:16:59 <zere> #topic carto style port
17:18:00 <pnorman> coastlines are apparently a non-issue, just requires some changes to the download script and some path changes in one location. that being said, I haven't actually tested, but I believe they should be okay
17:19:08 <pnorman> the other issue is I wanted to do some benchmarking for some potential speed enchancements. I've got access to benzene (osm.us server) which has a worldwide osm2pgsql db, but I'm not sure how to best benchmark
17:21:38 <zere> is it just a case of picking a few (meta)tiles of different densities and different zoom levels and timing how long they take to render?
17:22:02 <pnorman> I'll be looking at the speed of st_intersects vs the speed of && mainly, and I suspect that caching means that the technique of picking a city and rendering it won't work
17:23:26 <zere> or you'd have to pick a long list of places such that it wasn't possible for them all to be in cache at once?
17:23:47 <pnorman> ideally something representative of yevaud's load
17:25:50 <zere> yeah. i can give you a chunk of log, let me see what's available.
17:26:47 <iandees> comparing cached carto to cached xml would be useful
17:26:53 <pnorman> also if anyone happens to have the command line stuff to feed in a chunk of log
17:27:07 <apmon> pnorman: Do you care about uncached db access?
17:27:23 <apmon> Otherwise just run the test multiple times to ensure things are in cache
17:27:33 <apmon> and then compare the results after cache warmup
17:28:24 <pnorman> apmon: yes, I'll be doing that. the problem is that if I just test a small area vs testing random metatiles from cities around the world (but totalling the same area rendered) the cache performance will be different
17:29:22 <pnorman> although I guess given that ST_Intersects is _ST_Intersets AND && it wouldn't matter too much because the index usage would be the same
17:29:26 <zere> damn. i don't see anywhere that renderd logs what it has rendered. apmon: is there such a log?
17:29:57 <pnorman> It still would be nice to have an easy way to do these tests, including a real-world workload.
17:31:55 <apmon> It does spit that information out in its general debug log
17:32:01 <apmon> but not as a separate log file
17:32:14 <apmon> wouldn't be particularly hard to add that though
17:34:40 <apmon> If useful, please file it as an issue on github, and I'll see that I'll add it.
17:35:23 <pnorman> apmon: perhaps for now could you just send the shell script you used to test osm.xml vs carto?
17:35:23 <zere> would that be on stdout/stderr?
17:35:49 <apmon> syslog I think
17:36:33 <apmon> pnorman: I just used render_list with the all option and a bounding box
17:37:02 <apmon> but having a proper list of representative meta_tiles to benchmark would probably be quite helpful
17:37:43 <apmon> in some ways, the utility render_speedtest should handle this
17:37:52 <apmon> but I am not sure how well that currently works
17:40:31 <zere> pnorman: http://matt.dev.openstreetmap.org/renderd.log.gz is a 1h chunk of renderd debug output
17:40:49 <zere> at peak times, of course ;-)
17:41:28 <apmon> from yevaud?
17:43:11 <pnorman> apmon: got the bounding box (or better yet, the exact command used)?
17:43:38 <zere> apmon: yup
17:45:31 <apmon> pnorman: Looks like I used "./render_list --all -f -m carto -s ~/tiles/rednderd.sock --min-lon 8.1 --max-lon 8.6 --min-lat 48.9 --max-lat 49.2 -z 14 -n 4"
17:46:04 <apmon> Which is Karlsruhe, Germany and surrounding
17:46:13 <pnorman> thanks. i'll stick testing osm.xml vs current carto vs carto with st_intersects on my todo list
17:46:41 <pnorman> I understand the current blocker for deploying carto is a hardware one, right?
17:46:57 <apmon> zere: nice. Although it would probably in that case take several hours to do a test rendering of that
17:47:47 <apmon> I should probably add a mode to render_speedtest, that can read such a list and do a test rendering of that for benchmarking
17:51:02 <TomH> pnorman: not strictly a blocker, but easier to do everything at once
17:51:32 <pnorman> does yevaud have the capacity to deal with the (currently) slower carto style?
17:52:48 <zere> apmon: i'm sure the rendering characteristics are uniform enough that you could subset it randomly, or by line-slice and still have much the same characteristic... you'd need a world database, though.
17:54:24 <pnorman> one last carto item, after it's deployed it's going to need someone with cartographic design skills taking care of it
17:55:26 <zere> i don't know about capacity - we are pretty close to the line already. on the other hand, would it be better to have a style that people are able to contribute to and slower re-renders, or stick with this style - unchanged for the fear of XML horror - forever...?
17:56:27 <pnorman> zere: well, i guess it depends on how long we're talking for the hardware. if it's a matter of weeks I don't think a couple weeks or a month on osm.xml is too much. i wouldn't want to spend another 6 months on it
17:58:53 <apmon> the bottleneck on Yevaud seems to be CPU with regard to rendering?
17:59:34 <apmon> Would it be possible to just add an additional rendering server to take off some of the CPU load from Yevaud?
17:59:49 <apmon> perhaps one of the spare servers that are lying around anyway
18:00:17 <apmon> It wouldn't need any disks, and not much RAM.
18:01:42 <zere> i dunno about that - vmstat shows (on average) more processes waiting for i/o than running
18:03:08 <zere> and if the cpu usage is more in the database than mapnik, then having a low-disk, low-ram machine use the same DB on yevaud wouldn't be much help
18:03:28 <zere> the answer is always MOAR SSDs
18:03:44 <apmon> I think someone said it is about 50% / 50% renderd vs postgresql on CPU
18:04:18 <apmon> so if some of the 50% cpu usage of renderd can be offloaded, then postgresql would have more CPU available for itself
18:04:44 <apmon> Yevaud's SSD isn't directly the fastest anymore, so yes, there is opportunity for speed up there as well
18:06:31 <zere> cool. anything else to discuss?
18:08:46 <pnorman> that's it for me. apmon to lookat improving render_speedtest, pnorman to try benchmarking on benzene with existing tools, anything else?
18:10:50 <zere> i guess not. thanks all for coming & hope to see you next week :-)