How can I pull/checkout all or some builds/src.rpm from a repo like server:/* from specific dates via builds.opensuse.org in one go? I used to download sources from download.opensuse.org using a script.
However, the dates on some packages have changed since the migration, and my script now downloads broken packages alongside the new ones. I don’t want out-of-date packages with incompatible macros or an irresolvable status.
@coldboot I would suggest osc getbinaries not sure about specific dates. Not sure why you don’t just spin up your own private build instance and use that…
1 Like
I would suggest
osc getbinariesnot sure about specific dates. Not sure why you don’t just spin up your own private build instance and use that…
I tried ‘build,’ but it was noticeably slow at 200-250 rpms. OSC’s directory structure is complicated. By automating rpmbuild in podman (which I created with kiwi) and later purging unneeded packages, I am able to reduce the build time. I don’t want to create packages older than 2021-2020. My previous bash oneliner was downloading src.rpm matching date pattern, but with the new layout it doesn’t work on the main site; I can download from a mirror, but it pulls in everything.
I need to do some research on OSC to see if I can find the date of the package (changed status) and then try to automate the checkouts. I’m not used to OSC or extensive Linux build tools; I’ve been working on OpenBSD for years.
BTW, the openSUSE python-kiwi version has a bug in which kiwi-ng fails to create a compressed tarball for a docker image; I raised an issue on their github and it was fixed. I’ve created a package - Show home:coldboot / python-kiwi - openSUSE Build Service - and hope the bug fix is included in the next python-kiwi update.
What is your hardware setup? I have 24 cores/threads and 62GB of ram for this development desktop…
I’ve Intel Xeon E3-1245V3, 32 GB RAM, and a total of 10TB storage. Other box has Intel Xeon E3 1220 V2, 32 GB RAM and 8TB storage.
That should be fine then… not sure, used osc for so long, you could look at toolbox or distrobox for your build environment, maybe that would help?
I know with osc you can checkout a specific release/version, but not a date…
1 Like
I’ve had success with rpmbuild automation, I get a full log and can detect failed builds with grep -L “Wrote:” <spec.logfile>, if I find a way to verify the build results on OBS from cli then I will be able to minimize repetitive tasks and build more.
Yes, you can use osc to get any info on the build state, publish etc.
Consider using Qactus if you want a GUI…
1 Like
Great, I will give it a go. Found the API documentation – API Documentation - openSUSE Build Service
Do you mind if I occasionally DM you with my queries?
I’m on IRC (discord/matrix), there is a build service channel you can ping me there or just start a new thread 
1 Like
Alright, I will get on IRC then. Thank you.
You cannot. Only the latest successfully built version is ever kept.
You cannot. Only the latest successfully built version is ever kept.
The most recent successfully built version is not always built in the present day. I used to download packages published no later than 2020; this significantly reduced the build failure rate. The plan is to avoid wasting time on builds that may fail on this date.
curl "$URL" | sed -e 's/<[^>]*>//g' | grep -E '([0-31]-[Aa-Zz]-|2022|2021)' | awk -F '{print $1}' -F '\\.rpm' '{print $1".rpm"}' | sed -e '$ d' > "$LIST"
I have a feeling that you are trying to solve some problem that you have never described.
I have a feeling that you are trying to solve some problem that you have never described.
I’m getting src.rpm from a download.opensuse.org mirror, making hardened builds for some packages, and making a few packages relocatable (I know it is discouraged, but doing it as a learning experience). After evaluating the ‘build’ program and osc, I automated rpmbuild. The ‘build’ program downloads and installs BuildRequires, and once the build is complete, it purges all of them. Because of the directory structure, OSC usage is complicated for me. My automation is based on spec files, which help me with logs and allow me to run multiple loops. This exercise is intended for reducing overheads, ensuring fair usage of mirrors, not utilizing community resources for “experiments”.
My current problem is figuring out how to automatically eliminate failed builds on OBS when attempting to build via automation on a local machine.
Failed builds are not published to download.o.o. You still omit something that is obvious to you.
Yes, failed builds aren’t there but that doesn’t mean those packages from 2016 will build today on Tumbleweed. I attempted that on OBS but didn’t work.
The only packages you can be reasonably sure to build successfully for any given Tumbleweed snapshot are those from Ring 0. Any other package may fail whether it was last built in 2016, 2021 or 2022. The only way to check whether a given package will build successfully is to try to build it.
I suppose you could check whether last build attempt for a package failed. This will cover mass rebuilds or auto-rebuilds due to changed dependencies.
What migration do you mean?
1 Like
The layout at https://download.opensuse.org/repositories/network/openSUSE_Tumbleweed/x86_64/ has changed recently, earlier I could remove tags, create download list based on date and then use wget for downloading the src.rpm, I am right now exploring some other approach and will report back after 3-4 hours.
I’m not sure I understand Ring 0, and where I can find it. I’m going to build the packages; all I need to know is if I screwed up a failed package or if it’s already FUBAR without having to open a browser and search the stuff on the net.
I have a feeling that you are trying to solve some problem that you have never described.
I came up with a different solution to my problem. I read some OSC documentation, checked out a project, and then ran -
find . -type f -name "*tar*" -newermt 20210101 | grep -v ".osc" | xargs -I{} dirname {} | sort -u
I need to learn more about the OSC API. Normal console tools or pandoc appear to be unable to extract data from the remote webpage Show network - openSUSE Build Service
For the time being, I am able to achieve my goal, but this results in the download of 5GiB of files when I only require 2.2GiB; I would prefer not to burden the SUSE mirror infrastructure.