Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: How to extract OSM-files [ bz2] - in order to work on with the data?

Hybrid View

  1. #1

    Red face How to extract OSM-files [ bz2] - in order to work on with the data?

    hello dear linux-fans -good day!



    i just come back from the site

    Planet.osm - OpenStreetMap Wiki

    which has got some files that belong to the openstreetmap-Project.

    i downloaded a similar file that is derived from a German Mirrorsite - this one here:

    Index of /pub/misc/openstreetmap/download.geofabrik.de

    i downloaded the file called germany.osm,bz2


    according to this instruction

    type bzip2 -d planet.osm.bz2; or your OS may support double-click unpacking. See Wikipedia's list of compression programs.

    i did the following - but without success-,
    martin@linux-wyee:~/poi_abzug/geofabrik> bzip2 -d germany.osm,bz2;
    bzip2: Can't open input file germany.osm,bz2: No such file or directory.
    why - can anybody explain why i did not succeed?
    dilbert ;-)
    Wordpress-development - a Toolset: wpgear.org

  2. #2
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    27,124
    Blog Entries
    15

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    Quote Originally Posted by dilbertone
    hello dear linux-fans -good day!



    i just come back from the site

    'Planet.osm - OpenStreetMap Wiki'
    (http://wiki.openstreetmap.org/wiki/Planet.osm)

    which has got some files that belong to the openstreetmap-Project.

    i downloaded a similar file that is derived from a German Mirrorsite -
    this one here:

    'Index of /pub/misc/openstreetmap/download.geofabrik.de'
    (http://ftp5.gwdg.de/pub/misc/openstr....geofabrik.de/)

    i downloaded the file called germany.osm,bz2


    according to this instruction

    >
    > type bzip2 -d planet.osm.bz2; or your OS may support double-click
    > unpacking. See Wikipedia's list of compression programs.
    >



    > i did the following - but without success-,
    > martin@linux-wyee:~/poi_abzug/geofabrik> bzip2 -d germany.osm,bz2;
    > bzip2: Can't open input file germany.osm,bz2: No such file or
    > directory.
    >


    why - can anybody explain why i did not succeed?

    Hi
    Check the filename....
    Code:
    bzip2 -d germany.osm.bz2
    No , and no ;

    --
    Cheers Malcolm °¿° (Linux Counter #276890)
    SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 3.0.13-0.27-default
    up 0:13, 2 users, load average: 0.07, 0.11, 0.14
    CPU Intel i5 CPU M520@2.40GHz | Intel Arrandale GPU


  3. #3

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    Hello dear Malcom, good day,


    now it worked: see below..:

    Code:
    martin@linux-wyee:~/poi_abzug/geofabrik> bzip2 -d germany.osm,bz2
    i now have a large xml-file. Way too large to open it with Kwrite. The data contains all data (included restaurants) within Germany.

    now i want to use the tool, called Osmosis see - (Osmosis - OpenStreetMap Wiki).
    Osmosis is a great tool, an OSM data parser. (i gathered some information about the [detailed usage page](Osmosis/Detailed Usage 0.40 - OpenStreetMap Wiki) to find out more of how to use osmosis. Well, there may be other ways in osmosis to do it, Restaurants could be coded in OSM as a node or a way. Currently i am looking for the right repo that can be used for OpenSuse 12.1 32 Bit


    There may be other ways in osmosis to do it, Restaurants could be coded in OSM as a node or a way. currently i am looking for the right repo that can be used for OpenSuse 12.1 32 Bit

    Note: The following filter should leave me with a file that removes all ways and nodes that DO NOT have the value amenity=restaurant.
    osmosis --read-xml input.osm --tf accept-nodes amenity=restaurant \ --tf accept-ways amenity=restaurant \ --write-xml output.osm

    Afterwards i need to filter out all nodes that do not contain "amenity.restaurant".

    At the end i am tryin to get all into a CSV file ?

    What is aimed: i want to have the dataset - with full description and some with only a few data.. like the following

    PHP Code:
    6.9441000    50.9242000    [50677] (ItalCasa di Biase [Köln]
    6.9373600    50.9291800    [50674] (ItalAl Setaccio [Köln
    question: is this doable - can i work on osmosis to get an output like above...
    btw: what can i do. i need the full data.set. can i get this somewhere
    if i have all the position-data, is it possible to find the rest.

    Code:
    a. name of the street
    b. name of the town 
    .. is this possible...!?
    so that it looks like the following dataset;.

    PHP Code:
    10.5346100    52.1613600    [38300] (ChinWanbao Kommissstr.9 [Wolfenbüttel]
    13.2832500    52.4422600    [14167] (ItalLaPergola  Unter den Eichen 84d [Berlin]
    13.3177700    52.5062900    [10625] (ChinGood Friends  Kantstr.30 [Berlin

    Not sure if that works - but i am trying to do so,I don't think there's a particular program that directly exports OSM to CSV. Is there a particular reason you want it into a CSV file ?
    Well if i am using an application that supports imports of CSV, it might also support other file formats that .OSM data could be exported into.

    BTW at the moment i need to have the osmosis-tool that runs in opensuse: where to find the repository: i did a long search in the internet and finally found out this very intersting Mirror List - does this fit? Well i am not 100 sure if that is this the path that i need..? Note: The following filter should leave me with a file that removes all ways and nodes that DO NOT have the
    Code:
    value amenity=restaurant
    .

    That can be done with:
    PHP Code:
    osmosis --read-xml input.osm --tf accept-nodes amenity=restaurant \ --tf accept-ways amenity=restaurant \   --write-xml output.osm 
    Not sure if that works - but i am trying to do so. Hmm - i am not very sure if it works as expected. I don't think there's a particular program that directly exports OSM to CSV. Is there a particular reason you want it into a CSV file? Well if i am using an application that supports imports of CSV, it might also support other file formats that .OSM data could be exported into.

    any ideas.... on this tasks?! i am pretty new to all that GIS-stuff

    greetings
    dilbert ;-)
    Wordpress-development - a Toolset: wpgear.org

  4. #4
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    Am 15.04.2012 13:36, schrieb dilbertone:
    > now i want to use the tool, called Osmosis see - ('Osmosis -
    > OpenStreetMap Wiki' (http://wiki.openstreetmap.org/wiki/Osmosis)).
    > Osmosis is a great tool, an OSM data parser. (i gathered some
    > information about the [detailed usage page]('Osmosis/Detailed Usage 0.40
    > - OpenStreetMap Wiki'
    > (http://wiki.openstreetmap.org/wiki/O...Detailed_Usage)) to find out
    > more of how to use osmosis. Well, there may be other ways in *osmosis
    > *to do it, Restaurants could be coded in OSM as a node or a way.
    > Currently i am looking for the right repo that can be used for OpenSuse
    > 12.1 32 Bit

    Not sure if I understand really what your problem is. Osmosis is a java
    program, just download
    http://bretth.dev.openstreetmap.org/...sis-latest.tgz
    and extract, cd to the bin directory there and run ./osmosis

    --
    PC: oS 11.4 x86_64 | Intel Core i7-2600@3.40GHz | 16GB | KDE 4.8.2 |
    GeForce GT 420
    Eee PC 1201n: oS 12.1 x86_64 | Intel Atom 330@1.60GHz | 3GB | KDE 4.8.2
    | nVidia ION
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10 |
    xf86-video-geode

  5. #5
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    Am 15.04.2012 13:50, schrieb Martin Helm:
    > Not sure if I understand really what your problem is. Osmosis is a java
    > program, just download
    > http://bretth.dev.openstreetmap.org/...sis-latest.tgz
    > and extract, cd to the bin directory there and run ./osmosis
    >

    After reading a bit further about osmosis and what I understand about
    your task (read the xml filter some nodes and extract the filtered
    result into a csv file) I am not sure at all if osmosis is what you want
    or need.
    Whenever someone comes and asks how to transform an xml file into
    something else (with or without filtering) my answer is always: Look at
    xslt since this is what xslt is made for.
    Most common scripting languages also have good support for such xml
    processing tasks (python, perl ...) if you do not want to learn about xslt.

    --
    PC: oS 11.4 x86_64 | Intel Core i7-2600@3.40GHz | 16GB | KDE 4.8.2 |
    GeForce GT 420
    Eee PC 1201n: oS 12.1 x86_64 | Intel Atom 330@1.60GHz | 3GB | KDE 4.8.2
    | nVidia ION
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10 |
    xf86-video-geode

  6. #6

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    hello Martin,

    thx for the quick relpy...

    many thanks for the hints... i checked the prelimianries..

    Code:
    martin@linux-wyee:~/poi_abzug/geofabrik> java -version
    java version "1.6.0_24"
    OpenJDK Runtime Environment (IcedTea6 1.11.1) (suse-3.1-i386)
    OpenJDK Server VM (build 20.0-b12, mixed mode)

    ...and did the following steps according - DEsmosis/Installation - OpenStreetMap Wiki



    Code:
    linux-wyee:/home/martin # cd gis
    linux-wyee:/home/martin/gis # wget http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.tgz
    asking libproxy about url 'http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.tgz'
    libproxy suggest to use 'direct://'
    --2012-04-15 14:18:27--  http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.tgz
    Auflösen des Hostnamen »bretth.dev.openstreetmap.org (bretth.dev.openstreetmap.org)«... 128.40.168.103
    Verbindungsaufbau zu bretth.dev.openstreetmap.org (bretth.dev.openstreetmap.org)|128.40.168.103|:80... verbunden.
    HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
    Länge: 7683042 (7,3M) [application/x-gzip]
    In »»osmosis-latest.tgz«« speichern.
    
    100%[====================================================================================>] 7.683.042    414K/s   in 19s     
    
    2012-04-15 14:18:46 (391 KB/s) - »»osmosis-latest.tgz«« gespeichert [7683042/7683042]
    
    linux-wyee:/home/martin/gis # tar xvfz osmosis-latest.tgz
    osmosis-0.40.1/bin/
    osmosis-0.40.1/build/
    osmosis-0.40.1/config/
    osmosis-0.40.1/lib/
    osmosis-0.40.1/lib/default/
    osmosis-0.40.1/script/
    osmosis-0.40.1/script/contrib/
    osmosis-0.40.1/script/munin/
    osmosis-0.40.1/build/version.txt
    osmosis-0.40.1/changes.txt
    osmosis-0.40.1/config/plexus.conf
    osmosis-0.40.1/copying.txt
    osmosis-0.40.1/lib/default/aopalliance-1.0.jar
    osmosis-0.40.1/lib/default/commons-codec-1.5.jar
    osmosis-0.40.1/lib/default/commons-compress-1.2.jar
    osmosis-0.40.1/lib/default/commons-dbcp-1.4.jar
    osmosis-0.40.1/lib/default/commons-logging-1.1.1.jar
    osmosis-0.40.1/lib/default/commons-pool-1.5.4.jar
    osmosis-0.40.1/lib/default/jpf-1.5.jar
    osmosis-0.40.1/lib/default/mysql-connector-java-5.1.18.jar
    osmosis-0.40.1/lib/default/osmosis-apidb-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-areafilter-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-core-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-dataset-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-extract-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-hstore-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-pbf-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-pgsimple-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-pgsnapshot-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-replication-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-set-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-tagfilter-0.40.1.jar
    osmosis-0.40.1/lib/default/osmosis-xml-0.40.1.jar
    osmosis-0.40.1/lib/default/osmpbf-1.1.1-754a33af.jar
    osmosis-0.40.1/lib/default/plexus-classworlds-2.4.jar
    osmosis-0.40.1/lib/default/postgis-jdbc-1.3.3.jar
    osmosis-0.40.1/lib/default/postgresql-9.0-801.jdbc4.jar
    osmosis-0.40.1/lib/default/protobuf-java-2.4.1.jar
    osmosis-0.40.1/lib/default/spring-aop-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-asm-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-beans-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-context-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-core-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-expression-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-jdbc-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/spring-tx-3.0.6.RELEASE.jar
    osmosis-0.40.1/lib/default/stax2-api-3.1.1.jar
    osmosis-0.40.1/lib/default/woodstox-core-lgpl-4.1.2.jar
    osmosis-0.40.1/lib/default/xercesImpl-2.9.1.jar
    osmosis-0.40.1/readme.txt
    osmosis-0.40.1/script/contrib/CreateGeometryForWays.sql
    osmosis-0.40.1/script/contrib/apidb_0.6.sql
    osmosis-0.40.1/script/contrib/apidb_0.6_osmosis_xid_indexing.sql
    osmosis-0.40.1/script/contrib/dump_apidb.sh
    osmosis-0.40.1/script/contrib/replicate_osm_file.sh
    osmosis-0.40.1/script/fix_line_endings.sh
    osmosis-0.40.1/script/munin/README
    osmosis-0.40.1/script/munin/osm_replication.conf
    osmosis-0.40.1/script/munin/osm_replication_lag
    osmosis-0.40.1/script/pgsimple_load_0.6.sql
    osmosis-0.40.1/script/pgsimple_schema_0.6.sql
    osmosis-0.40.1/script/pgsimple_schema_0.6_action.sql
    osmosis-0.40.1/script/pgsimple_schema_0.6_bbox.sql
    osmosis-0.40.1/script/pgsimple_schema_0.6_linestring.sql
    osmosis-0.40.1/script/pgsnapshot_and_pgsimple.txt
    osmosis-0.40.1/script/pgsnapshot_load_0.6.sql
    osmosis-0.40.1/script/pgsnapshot_schema_0.6.sql
    osmosis-0.40.1/script/pgsnapshot_schema_0.6_action.sql
    osmosis-0.40.1/script/pgsnapshot_schema_0.6_bbox.sql
    osmosis-0.40.1/script/pgsnapshot_schema_0.6_linestring.sql
    osmosis-0.40.1/script/pgsnapshot_schema_0.6_upgrade_4-5.sql
    osmosis-0.40.1/script/pgsnapshot_schema_0.6_upgrade_5-6.sql
    osmosis-0.40.1/bin/osmosis
    osmosis-0.40.1/bin/osmosis-extract-apidb-0.6
    osmosis-0.40.1/bin/osmosis-extract-mysql-0.6
    osmosis-0.40.1/bin/osmosis.bat
    linux-wyee:/home/martin/gis # cd osmosis-0.39
    bash: cd: osmosis-0.39: Datei oder Verzeichnis nicht gefunden
    linux-wyee:/home/martin/gis # chmod a+x bin/osmosis
    chmod: Zugriff auf „bin/osmosis“ nicht möglich: Datei oder Verzeichnis nicht gefunden
    linux-wyee:/home/martin/gis # bin/osmosis
    hmm guess some trouibles occurred here... i cannot understand whats going on here...
    dilbert ;-)
    Wordpress-development - a Toolset: wpgear.org

  7. #7
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    Am 15.04.2012 14:36, schrieb dilbertone:
    > [snip]
    > osmosis-0.40.1/bin/osmosis.bat
    > linux-wyee:/home/martin/gis # cd osmosis-0.39
    > bash: cd: osmosis-0.39: Datei oder Verzeichnis nicht gefunden


    I hope you notice that you have to cd to osmosis-0.40.1 and not to
    osmosis-0.39?

    --
    PC: oS 11.4 x86_64 | Intel Core i7-2600@3.40GHz | 16GB | KDE 4.8.2 |
    GeForce GT 420
    Eee PC 1201n: oS 12.1 x86_64 | Intel Atom 330@1.60GHz | 3GB | KDE 4.8.2
    | nVidia ION
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10 |
    xf86-video-geode

  8. #8
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default AW: Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    The following completely trivial xslt will do all the extraction stuff

    Code:
    <xsl:stylesheet version = '1.0'
        xmlns="http://www.w3.org/1999/xhtml"
        xmlns:xsl='http://www.w3.org/1999/XSL/Transform'>
    
        <xsl:output method="text" encoding="UTF-8"/>
        <xsl:template match="/">
        
            <xsl:for-each select="/osm/node/tag[@k='amenity' and @v='restaurant']">
                <xsl:value-of select="../@id"/>
                <xsl:text>&#x09;</xsl:text>
                <xsl:value-of select="../@lat"/>
                <xsl:text>&#x09;</xsl:text>
                <xsl:value-of select="../@lon"/>
                <xsl:text>&#x09;</xsl:text>
                <xsl:for-each select="../tag[@k='name']">
                    <xsl:value-of select="@v"/>
                </xsl:for-each>
                <xsl:text>&#x0A;</xsl:text>
            </xsl:for-each>
        </xsl:template>
    
    </xsl:stylesheet>
    I gave it the name restaurants.xslt, adjust it to your needs (additional fields, different formatting ...)

    You can run that directly on your bz2 compressed file with

    Code:
    bzcat germany.osm.bz2 | xsltproc restaurants.xslt - > restaurants,csv
    (I just tested with a 10MB subset of the data which takes around 2 seconds, so expect a full run on your multi GB file to take a long long time).

    No need for such an task to install some fancy extra software (if xsltproc is not in your system it is the package libxslt1).

  9. #9

    Default Re: AW: Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    hello dear Martin

    - many thanks for the quick reply and your great support - yet another great help you supply here to my trials...

    you help me to take bigger steps...

    btw: i have had a closer look at the system :


    firstly i had a closer look if i have the xslt-processor...

    Code:
    linux-wyee:/home/martin/gis # xsltproc -version
    Using libxml 20708, libxslt 10126 and libexslt 815
    xsltproc was compiled against libxml 20708, libxslt 10126 and libexslt 815
    libxslt 10126 was compiled against libxml 20708
    libexslt 815 was compiled against libxml 20708
    linux-wyee:/home/martin/gis # ^C

    so far so good.... now i tried to replicate your big step - handling the whole file and making a big step in processing the data...


    Code:
    linux-wyee:/home/martin/gis/german_poi # bzcat germany.osm.bz2 | xsltproc restaurants.xslt - > restaurants,csv
    warning: failed to load external entity "restaurants.xslt"
    cannot parse restaurants.xslt
    linux-wyee:/home/martin/gis/german_poi #
    hmm - why do i get cannot parse restaurants.xslt.... ?!

    i try to have a closer look at it .. Perhaps i can find out what causes this error?

    Look forward to hear from you..
    dilbert ;-)
    Wordpress-development - a Toolset: wpgear.org

  10. #10
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default Re: How to extract OSM-files [ bz2] - in order to work on with thedata?

    Check if the file you created really has the name restaurants.xslt and
    that it is in your current working directory
    Code:
    ls -l restaurants.xslt
    from where you try to run the conversion command because the error
    indicates one of both requirements is not fulfilled.

    --
    PC: oS 11.4 x86_64 | Intel Core i7-2600@3.40GHz | 16GB | KDE 4.8.2 |
    GeForce GT 420
    Eee PC 1201n: oS 12.1 x86_64 | Intel Atom 330@1.60GHz | 3GB | KDE 4.8.2
    | nVidia ION
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10 |
    xf86-video-geode

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •