Can't expand headers within `<>`

[ishandutta2007]

I believe the code is designed to expand headers within "" only.
For example when I am trying to amalgamate this code primecount api,
since in #include <primecount.hpp> the header name primecount.hpp is enclosed by <> it fails to pull in the required code.
Ideally we should use a lookup table to check if a header is native c++ header or not . Dont you think so?
Please do update if you can get quom to work with the above repo.

PS: This ticket has nothing to do with my other ticket. Now I am on macOS and I dont have that issue here.

[Viatorus]

Yes, this is currently not possible but your repo would never work reliable. I think you should rework your includes.

For example this one:

https://github.com/kimwalisch/primecount/blob/master/include/gourdon.hpp

#include <int128_t.hpp>
#include <print.hpp>

#include <stdint.h>

If you use <...>, a compiler assumes it is a system header.

By default, g++ for example will first try to find this header file in this order:

• /usr/include/c++/11
• /usr/include/x86_64-linux-gnu/c++/11
• /usr/include/c++/11/backward
• /usr/lib/gcc/x86_64-linux-gnu/11/include
• /usr/local/include
• /usr/include/x86_64-linux-gnu
• /usr/include

If it didn't find the header there, it will start the search relative to the file with the #include.

P.S. <stdint.h> should be included through (as every c header)

[ishandutta2007]

If it didn't find the header there, it will start the search relative to the file with the #include.

Yes thats what the logic should be, if that logic is already coded why does then "" or <> make a difference ?

I think you should rework your includes.

We should try to make quom work for as many repos as possible. Changing an entire repo everytime is definitely not a scalable option, if we could upgrade quom with small changes. Among all amalgamate
tools I came across yours seem to be the most general . I am okay to send you a pull request if you dont have bandwidth, but first you should help me understand.

PS:- I was surprised to find not many advanced fully working code amalgamator are there. One man project like Quom is among the best we have. I guess software industry don't need it much , usually it is the reverse they often need, hence no one bothered about moking one. But I would need it a lot . I need to extract code out into a single file from many big repos. Currently I do it manually , it take days of effort.

[Viatorus]

why does then "" or <> make a difference ?

• "" starts with a relative search and then searches inside the system includes
• <> starts with a search in the system includes and then a relative searches

So if you use <> for your relative include and the system e.g. provides a <min.hpp>, this would be used instead your files.

We should try to make quom work for as many repos as possible.

Yes, as long as this project structure is a common one doesn't have conceptional issues.

And yours has some issues.

• your include directory contains
  • the public API header primecount.hpp and primecount.h
  • and the private headers

1. Private headers do not belong inside the include folder because typically, the folder will be copied to the system include directory and every header would be public available (same for package manager like conan).
2. Your include folder requires (because it is common best practice) a subfolder, so a include of your library would be #include <primecount/primecount.hpp>

I think, if you would fix at least 1.) and use relative includes, you would have a cleaner public API without introducing a breaking change and can use quom out of the box.

[ishandutta2007]

And yours has some issues.

None of the big repos are mine . If I had written these for my purpose, I would write the required code in a single file directly not needing something like quom at all. As long as it builds successfully they won't consider it an issue. I can't just ask all big repos like the above one to change themselves as per quom, quom has to change to support as many of the popular repos as possible. And why would they even listen to you if you just call their project as "conceptional issue"-ed when hundreds of people are using those successfully everyday.

[ishandutta2007]

Yes, as long as this project structure is a common one doesn't have conceptional issues.

I tried a couple of other popular repos too. Couldn't get quom to work for any till now. Can you make it work for this repo ? Say this example https://github.com/flintlib/flint2/blob/trunk/examples/crt.cpp
I couldn't quom here either. For exact issue I may create a separate ticket if you want. But first tell me does this FLINTLIB repo fall under the common structure that you are talking about ?
You can assume I have taken the crt.cpp out of the /examples folder to the root of the project before running quom on it, if that satisfies the case you are making.

[ishandutta2007]

These are the repos I generally work with. I am not sure for how many of them quom will work out of the box
https://en.cppreference.com/w/cpp/links/libs
https://en.wikipedia.org/wiki/List_of_numerical_libraries#C++

[ishandutta2007]

does this FLINTLIB repo fall under the common structure that you are talking about ?

Did you manage to get it working?

Seems like quom has only been tested on toy projects. To make quom into something of practical utility we should test it against top 10 or top 20 C/C++ open-source libraries. I need amalgamation quite a lot. I do it manually spending days. I have been wanting to start a amalgamation tool for a while. I would definitely like to help you take quom to something of practical utility provided I have your support.

[ishandutta2007]

?

[ishandutta2007]

Can you kindly reply. It is okay to start a project aimed at toy projects only or keeping community only to self etc.. Different people have different motivation in starting open-source projects. Not all repos have the same goal. Unless you speak up we wont get to know what you intend to do with quom.

[Viatorus]

None of the big repos are mine

I was only focusing on primecount. I didn't pay attention to any other library. I assumed you are the author of this library primecount.

As long as it builds successfully they won't consider it an issue.

Well, it may be no issue if you only build primecount alone for itself but the code structure design has an issue and if any other project does the same and you combine them nothing works as expected anymore.

And why would they even listen to you if you ... when hundreds of people are using those successfully everyday

A library (like primecount) should not only provide a good algorithm/implementation but also a safe deployment and distribution strategy.

tested on toy projects.

How dare you to insult anybody's project a toy project?

But first tell me does this FLINTLIB repo fall under the common structure that you are talking about ?

Imho the project structure is still a little mess . Please take a look at libraries like boost, abseil, opencv and poco where I take my orientation from.

Did you manage to get it working?

I tested it and it fails on several steps. Most are include bugs because many developer do not understand the difference between "" and <>.

1. flint.h includes the system include limits.h with "", which is wrong. Quom cannot find this include in the library repo and fails. At all other places in flint, <limits.h> is used. This one is maybe a typo?! Changed to <>.

2. flint-config.h isn't found. Okay this would be something CMake would automatic configure. Quom does not interact with CMake. I generate this file manually.

3. Than it fails for finding "gc.h" which is an external library (maybe this one? https://github.com/doublec/gc/blob/master/gc.h - I don't know) . Same as system includes, external libraries must be included with <>. Changing to <>.

4. Than fmpz-conversions.h - again a CMake generated file. I generate this file manually.

Done. 11350 lines of code. And it compiles: https://godbolt.org/z/1ErMqMG56 (if GCC < 11 is used because of a conversion bug).

It is okay to start a project aimed at toy projects only or keeping community only to self etc.. Different people have different motivation in starting open-source projects. Not all repos have the same goal.

I think the overall misunderstanding that happend here is that I assumed you are a library author and you assumed Quom is a tool other third party libraries can be unified by just running Quom over it.

Quom is designed for library authors who want to simplify the distribution of there multi-file project. So other can simply download one file and include it in there code base. This need is mainly because of the lack of a widely used package manager for C++. I never tested Quom against any existing library. But there are some libraries out there. One with many stars I now of is: https://github.com/hanickadot/compile-time-regular-expressions

Your need is something different - I assume for performance reason?

I don't think Quom would be ever capable to support libraries like flint out of the box if special configuration/build steps are necessary. Quom follows an include/header/source pattern. If this isn't given or not consistent - Quom will fail.

There are also other cases where a successfully Quom generated file will never compile. If two source files do have an identical named internal symbol (like foo.c and bar.c both defines a static int x = 0;).

I am open to support more projects - projects that uses <> for including their own headers which is not wrong if done right. E.g. like boost is doing it: https://github.com/boostorg/beast/blob/develop/include/boost/beast.hpp#L15 - it could be possible to add options/paths to support this kind of projects.

[ishandutta2007]

Thanks for the reply.

Imho the project structure is still a little mess . Please take a look at libraries like boost, abseil, opencv and poco where I take my orientation from.

These names that you mentioned are also among top 20-30 open-source libraries as well. Does quom already support these? It would be great if it's tested against these and ensured support for these to start with. If they have been already tested we should call that out in the docs or preferably somewhere in the examples or tests folder.

How dare you to insult anybody's project a toy project?

it's definitely not about insult, its a common terminology in software industry, we often use the terms "toy test data" or "mock test data" to mean something that has not been tested on real production or production quality data yet.

I assumed you are a library author and you assumed Quom is a tool other third party libraries can be unified by just running Quom over it.

not looking to unify the entire library but only the code that I need.

Quom is designed for library authors who want to simplify the distribution of there multi-file project.

I kind of got that from your earlier reply. Running quom on own project doesn't really appeal to me. It may have some small users as you mentioned. My view is as a project matures generally the progression is from flatter structure to more hierarchical structure. You are talking about simplified distribution vs my requirement of extraction of a subset from complex hierarchical project. Do yo know of any other amalgamator tool whose philosophy is more inclined towards mine?

I tested it and it fails on several steps.

You have more patience than me to deal with those errors, and of course better understanding of quom.
Distributing the same code in a flatter structure doesn't save space, on top of that complicates readability as well. So Someone who adds an entire library as single file to their library is actually doing more harm than declaring another dependancy. So I am sill not convinced with this use case.
Anyways my use case is primarily to save source code size. And on the hosts I wish to run the code, I don't have the liberty to install any third party library either . In most case I just need to extract less than 1% of the code from those large open-source libraries. As you found out for the crt.cpp case that you just did, you extracted out 11K lines out of 890K lines.
Now that we both have understanding of each others motivation can you suggest some better approach for my use case.

[Viatorus]

These names that you mentioned are also among top 20-30 open-source libraries as well. Does quom already support these?

Never tested and it is also not the scope of this project. Boost uses <> for his own header inclusion (which is fine) but Quom does not yet support this.

If they have been already tested we should call that out in the docs or preferably somewhere in the examples or tests folder.

Even through Quom could claim at some point, these libraries can be amalgamate minor changes in the sources (duplicated internal symbol) could break all.

not looking to unify the entire library but only the code that I need.

So not the scope of this project.

Do yo know of any other amalgamator tool whose philosophy is more inclined towards mine?

No and I never heard of such a use case like yours.

Now that we both have understanding of each others motivation can you suggest some better approach for my use case.

No, not really. I don't know why you have to save source code size. Do you mean binary code size? Can't you link statically or something like that?

[ishandutta2007]

binary code size?

No it's source code size, you only have mentioned that the amalgamated file you generated for me was 11K lines while the original repo size was 890K lines. In fact this shows that your approch too doesn't pull in the entire 890K lines either but only the functons that are being called from the initiating program. So your claim that you are amalgamating the entire library is not true.

why you have to save source code size.

Because I don't have money to buy so much of hard disk space. :D

Can't you link statically or something like that?

No I can't, as I said the production environment I work with has lots of restrictions. I have strict limitation on source code size, runtime memory, number of source files etc.

[ishandutta2007]

In fact gcc -E almost does the job for me, only that I don't need to print the codes from system headers.
So My idea of building an amalgamator is just to tweak gcc -E such that it suppresses the codes from system headers.
As long as gcc compiles successfully so would gcc -E and so would gcc -E -my_new_option.
In such case we need not worry about structure, format, tokenizer, parenthesis or <>, interaction type with CMake, duplicated symbols/triplicated symbols or any other BS whatsoever.

Infacf preprocessor(ie gcc -E) pulls in everything but amagamator like quom doesn't. Just that gcc -E isn't choosy about what to amagamate. To keep track I am linking my stackoverflow question here.

[ishandutta2007]

your approch too doesn't pull in the entire 890K lines either but only the functons that are being called from the initiating program. So your claim that you are amalgamating the entire library is not true.

I thought you would at least respond to this since it does more like what I expected an amalgamator to do that what you intended to build it for.

Anyways nevermind lets come back to the issue under the ticket, I have reworked all the imports here (check my fork only) . Let me know if it now satisfies the project structure you desire.