Terms: BinaryBuilder.jl Linux SimpleITK JuliaBinaryWrappers PkgTemplates Medimage.jl

LLM used : LLama 3 70B via Groq cloud, temp:1, max_tokens:1024
system_prompt: You are a sophisiticated medical imaging data scientist, equipped with vast array of knowledge around scientific computing and medical imaging, specifically around julia language , dicom and nifti datasets and the SimpleITK package. Please respond to the given follow up questions by providing accurate responsed, in order to be maximally helpful. Please help me write a julia package around SimpleITK , by helping me building binaries for it using binarybuilder.jl

Getting Started

Install the language specific binaries for a particular library, SimpleITK offers ".whl" installable binaries specific for python and other langs such as java, which further can be installed within the relevant language's kernel using a compatible package manager within that language for eg: mamba, conda or pip for python. In order to build c++ binaries for SimpleITK, though the organization doesn't list any binaries that could provide a readable and callable interface via C++. (The ideal binaries are in a file format called ".so", which upon loading in a julia program via

Julia discussion around building binaries for wrappers

Using Libdl
Libdl.dlopen(sample_binary.so)

will read the binary and will provice a c++ callable interface within julia for any functions exported within the read binary, and you can further wrap any functions from such binary within your julia package via

ccall()

The above function, allows a native C++ interface within the julia kernel.

Project Flow:

Package X requires dependency y
Dependency y doesnt have native binaries with a c++ interface
Building the y binaries a dependency for pkg x via preferred build tools
Loading the binary within pkg x and using ccall() to wrap the library's exported functions.
The build binaries and the pacakge creations, only benefits the user, as the binaries are build on the host system locally.

Usage of BinaryBuilder.jl

BinaryBuilder documentation

BinaryBuilder.jl allows to build binaries of dependecy libraries as native binaries for most of the supported platforms. The project flow in BinaryBuilder.jl involves, a build_tarball.jl script which initializes the flow of acquiring the source project and building it for a specific platform. This is further automated with the help of the BinaryBuilder wizard (run_wizard()), that provides a seamless menu based toolkit to build binaries without tinkering much with the build_tarball.jl.

The resulting binary will not just be a binary but a wrapper julia package or a JLL package, that needs to be uploaded over at the

Julia Binary Wrappers

in order to make the wrapped binary accessible as a package to the users, who can now call the functions from the pacakge and add the package as a normal julia package.

BinaryBuilder wizard video tutorial :

Binary Builder wizard tutorial

using BinaryBuilder
BinaryBuilder.run_wizard()

NOTE: BinaryBuilder.jl runs on linux x86_64 and macos_x86_64(intel) platforms, with windows support in active development. Both windows and macos hosts require docker support, as per the directions of BinaryBuilder docs.

Facilitating MedImage.jl package construction by providing SimpleITK binaries as a dependency for wrapping Image IO functionality

The intention behind this quest, was to facilitate the development of the MedImage.jl package (under active development), in order to provide a toolkit in julia that can seamlessly work with a bunch of medical imaging file formats such as nifti, dicom and hdf5 out of the box in terms of storing spatial metadata from such files.

Task 01: Building SimpleITK from source using BinaryBuilder.jl

Comprehensive guide put forward by SimpleITK:

Building SimpleITK

Comprehensive SimpleITK visual build guide

#Specifications : julia-kernel version : 1.10.3
using Pkg
Pkg.add("BinaryBuilder.jl")

Using the BinaryBuilder.jl wizard within a julia repl

Using BinaryBuilder.jl
BinaryBuilder.run_wizard()
#options
#option for building for platforms
Decided to build only for linux systems
#option for fetching the source via archive or git repo
https://github.com/SimpleITK/SimpleITK/releases/download/v2.3.1/SimpleITK-2.3.1.tar.gz
#option for additional binary dependencies
#option for changing compiler option
Changed the compiler option to select gcc for a version of 13.1 and above
#option for naming the resulting project
SimpeITK_bin
#option for maintaining the project version
2.3.1 (syncing up with the SimpleITK versioning)

#option for executing build commands via bash shell
based on the SimpleITK build instruction did the following,
mkdir SimpleITK-build
cd SimpleITK-build
cmake ../SimpleITK-2.3.1/SuperBuild
make

Error 1 : reference :

Info about gcc version update from here (fixed with wizard option to change compiler options in binarybuilder)

Error 2: Reference : with solution

SimpleITK build error with gcc 13.1 version due to a mathematical class enum definition.
fixed by introducing the the following code in SimpleITK-build directory,

SimpleITK-build->ITK->Modules->Filtering->MathematicalMorphology->include->itkMathematicalMorphology.h

added the following line
#include <cstdint>

added the std:: in line 41 of the file
enum class Algorithm : std:: uint8_t

Above is irrelevant, since BinaryBuilder run_wizard only creates a suitable build_tarball.jl file after testing the build process, alas the following final build_tarball.jl created a jll package for SimpleITK.

julia ./build_tarballs.jl --debug --verbose --deploy="divital-coder/LibSimpleITK_jll.jl"

name = "SimpleITK"
version = v"2.2.0"

# Collection of sources required to complete build
sources = [
    ArchiveSource("http://github.com/SimpleITK/SimpleITK/releases/download/v$(version)/SimpleITK-$(version).tar.gz", "b07bb98707556ebc2b79aac22dc14950749f509e5b43da8043233275aa55488a")
]

# Bash recipe for building across all platforms
script = raw"""
cd $WORKSPACE/srcdir
mount -t tmpfs -o size=12G tmpfs /workspace/srcdir
cd ..
cd ..
cd workspace/srcdir
wget https://github.com/SimpleITK/SimpleITK/releases/download/v2.2.0/SimpleITK-2.2.0.tar.gz
tar -xvzf ./SimpleITK-2.2.0.tar.gz 
mkdir SimpleITK-build
cd SimpleITK-build/
cmake -DCMAKE_INSTALL_PREFIX:FILEPATH=/workspace/destdir -DCMAKE_BUILD_TYPE:STRING=RELEASE -DBUILD_SHARED_LIBS:BOOL=ON ../SimpleITK-2.2.0/SuperBuild
make -j${nproc}
rm -rf /workspace/srcdir/SimpleITK-build/ITK-build/lib/cmake
cd SimpleITK-build
make install
cp /workspace/srcdir/SimpleITK-build/ITK-build/lib/* /workspace/destdir/lib
cd /workspace/destdir/share
mkdir licenses
cd licenses
mkdir SimpleITK
cd SimpleITK
cp /workspace/destdir/share/doc/SimpleITK-2.2/* ./ 
cp /usr/lib/libgcc_s.so.1 /workspace/destdir/lib
logout
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = [
    Platform("x86_64", "linux"; libc = "glibc")
]


# The products that we will ensure are always built
products = [
    LibraryProduct("libSimpleITKIO-2.2", :libSimpleITKIO),
    LibraryProduct("libSimpleITKRegistration-2.2", :libSimpleITKRegistration),
    LibraryProduct("libSimpleITKCommon-2.2", :libSimpleITKCommon),
    LibraryProduct("libSimpleITK_ITKCommon-2.2", :libSimpleITK_ITKCommon),
    LibraryProduct("libSimpleITK_ITKRegistrationCommon-2.2", :libSimpleITK_ITKRegistrationCommon),
    LibraryProduct("libSimpleITK_ITKSuperPixel-2.2", :libSimpleITK_ITKSuperPixel)
   ]

#exposing itk library products will result in unsatisfied build products,
#seems like the helloworld c++ from simpleitk ran with the following command
#before this do this
#cp /usr/lib/csl-glibc-x86_64/libgcc_s.so.1 /workspace/destdir/lib
#cp /usr/lib/csl-glibc-x86_64/libstdc++.so.6 /workspace/destdir/lib
#g++ -I./include/SimpleITK-2.2 -o hello_world test_sitk.cpp -L./lib -lSimpleITKCommon-2.2 -lSimpleITKIO-2.2 -lSimpleITKRegistration-2.2 -lSimpleITKBasicFilters0-2.2 -lSimpleITKBasicFilters1-2.2 -lSimpleITK_ITKImageFeature-2.2 -lSimpleITK_ITKImageCompose-2.2 -lSimpleITK_SimpleITKFilters-2.2 -lSimpleITK_ITKImageIntensity-2.2 -lSimpleITK_ITKImageSources-2.2 -lSimpleITK_ITKThresholding-2.2

#u will get errors based on what was needed to be linked , with the shared object, check and link

# Dependencies that must be installed before this package can be built
dependencies = Dependency[
]

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; julia_compat="1.6", preferred_gcc_version = v"8.1.0", verbose=true)

The above is a build_tarball.jl file for building SimpleITK-v2.2.0 with gcc8, i tested the above g++ (gcc based compiler) with the following c++ program , embedding Sitk functionality :

#include "SimpleITK.h"
#include <iostream>
using namespace std;
namespace sitk = itk::simple;
int main(int argc, char* argv[]){
if(argc < 2)
{
  cerr<< "Usage :" << argv[0] << " <input_nifti_file>" << endl;
  return EXIT_FAILURE;

}


  sitk::ImageFileReader reader;
  reader.SetFileName(argv[1]);
  sitk::Image image = reader.Execute();

  cout << "Image Information : "<< std:: endl;
  cout << "Size :" << image.GetWidth() << "x" << image.GetHeight() << "x" << image.GetDepth() << endl;
  cout << "Origin :";
  for (const auto& val : image.GetOrigin()){
cout << val << " ";
  }
cout << endl;

cout << "Spacing :";
for(const auto& val : image.GetSpacing()){

cout << val << " ";
}
cout << endl;


cout << "Direction : ";

auto direction = image.GetDirection();
size_t dimension = image.GetDimension();

for (size_t i=0; i < direction.size(); ++i){

if (i > 0 && i % dimension == 0){

cout << " : ";
}
cout << direction[i] << " ";
}
cout << endl;




return EXIT_SUCCESS;

}

Further this code when compiled using the following command :

g++ -I./include/SimpleITK-2.2 -o read_nifti_file read_file.cpp -L./lib -lSimpleITKCommon-2.2 -lSimpleITKIO-2.2 -lSimpleITKRegistration-2.2 -lSimpleITKBasicFilters0-2.2 -lSimpleITKBasicFilters1-2.2 -lSimpleITK_ITKImageFeature-2.2 -lSimpleITK_ITKImageCompose-2.2 -lSimpleITK_SimpleITKFilters-2.2 -lSimpleITK_ITKImageIntensity-2.2 -lSimpleITK_ITKImageSources-2.2 -lSimpleITK_ITKThresholding-2.2

In the above code , we have to list all the SimpleITK_ITk based shared objects files from the destdir lib directory, then the code will be compield with no errors/

finally run the program with an input file from the medimage/test_data

./read_nifti_file ./MedImage.jl/test_data/volume-0.nii.gz

and voila , we got a list of file information based on our origin code.

Image Information :
Size :512x512x75
Origin :-172.9 179.297 -368
Spacing :0.703125 0.703125 5
Direction : 1 0 0 : 0 -1 0 : 0 0 1

Hit and Trial, deemed beneficial

So apparently, building SimpleITK and distributing its binaries as shared objects with a build_tarballs.jl file over onto at yggdrasil, wasn't really the right move. The underlying wrapper building cmake configuration for SimpleITK is superbuild, which basically was configuring the build environment to install 3rd party dependencies, which were supposed to be added as jll packages in the dependencies section of the build_tarball.jl file. Further the most notable thing here, is the underlying ITK dependency being built during the SimpleITK build. Adhering to a package ecosystem, the following workflow should be in place :

Building ITK as a jll binary wrapper

Building SImpleITK as a jll binary wrapper witk ITK_jll wrapper as dependency

Since ITK would have sufficed our needs, there isn't any value in building SimpleITK anymore. Rather, a standalone ITK wrapper package in julia is the actual need here.

Building a wrapper in julia

In order to build a wrapper for c++ libraries in julia, the following workflow makes the most sense.

Building distributable binaries of the C++ library needed to be wrapped as a jll package with a PR to the yggdrasil. Eg: Xyce

Building distributable C++ Wrapper code (C++ part of CxxWrap.jl) binaries with the Binary jll package as a dependency, with the help of a PR to the yggdrasil. Eg: XyceWrapper

Building a new julia .jl package with the Binary Wrapper jll package as a dependency, to wrap the exposed functions as modules using the julia part of the CxxWrap.jl, and providing it within the julia registery. Eg: Xyce.jl

Current progress of ITK with a PR to the yggdrasil :

https://github.com/JuliaPackaging/Yggdrasil/pull/8795

Failure history with PR log to the yggdrasil

Here are all the attempts in trying to get ITK package to the yggdrasil after progressively better understanding around the BinaryBuilder.jl workflow and contribution routines.

https://github.com/JuliaPackaging/Yggdrasil/pulls?q=author%3Adivital-coder

Building binaries as dependencies for packages in julia for MedImage.jl package construction