For future development it's probably wise and might even be required to restructure the project, so that for example Delphi compiler and/or Free Pascal compiler keeps functioning properly.
The suggestion is to do the following:
Create folders for each UFilename.Pas in the Core Folder.
(Core Folder should be renamed to Library, so that PascalCoin Core can be considered a library to be used by other applications)
Example:
UBlockChain.pas -> create folder called BlockChain
UOpTransaction.pas -> create folder Transactions
Then split the above file into multiple units inside their folders.
## Expected Behavior
Better structuring of code.
## Current Behavior
Messy, long scroll times when editing, mess unnecessary/irrelevant code to be scrolled passed, time wasted searching/seeking etc.
## Possible Solution
Re-structuring.
## Steps to Reproduce
Examine current state of code.
## Context (Environment)
While moving around compiler crashed and ran out of memory.
Later IDE crashes on loading recovery file.
This is a clear sign that Delphi does not like the current PascalCoin code structure.
Hopefully Delphi will improve in the future, for now PascalCoin restructure is wise ;)
## Possible Implementation
Definetly possible to do this. Molina will need some time getting used to new structure. This is a slight disadventage, ultimate his development time will probably speed up.
While restructuring (UAccounts.pas) and pressing F9 another out of memory situation occured with Delphi XE Seattle.
Probably caused by more circular references, trying to reduce those.
Meanwhile there are some suggestions how to avoid Delphi XE from going out of memory:
http://support.embarcadero.com/article/44279
Some interesting suggestions are to:
1. Disabling code/error insight features (not sure if this helps/works) but can be tried.
2. Using MS Build to force the compiler to run in it's own process. (Seems very good suggestion, haven't tried it yet).
3. Another possible cause was that produce "map" file was on, not sure if this is increasing the problem.
4. Reducing circular references.
Many other users of Delphi XE have reported these problems.
PascalCoin isn't really that large, Delphi XE seems to be using 200 MB.
Thus I suspect it's a weird compiler bug which could be caused by circular references and the compiler ends up going into some kind of loop and allocating because of this huge ammounts of memory.
I suspect and expect that once the circular references are removed, the out of memory situations will disappear.
I can see the memory usage spike up from 200 MB to 2.7 GB.
Under normal circumstances memory usage for Delphi IDE/compiler is 200 to 300 MB or so, so definetly a Delphi XE shortcoming/bug.
I will first try other XE versions to see how severe this issue is.
Delphi 2010 has issues on Windows 7
https://sourceforge.net/projects/dzeditorlineendsfix/
https://support.microsoft.com/en-us/help/2993651/ms14-045-description-of-the-security-update-for-kernel-mode-drivers-au
https://docs.microsoft.com/en-us/security-updates/SecurityBulletins/2014/ms14-045
Solution is to rename this file whenever this happens to correctly startup Delphi 2010, examining further.
Delphi 2010 is not affected by circular reference bug, it does correctly display it:
[DCC Fatal Error] UAccounts.pas(23): F2047 Circular unit reference to 'UAccounts'
Delphi XE7 also not affected, might be because code/error insight is off I usually code without that, didn't check though.
For now it's safe to conclude that Delphi XE Seattle has a circular unit detection failure bug.
Now I am very curious if this bug may be caused by this future or not and if there is a fix for this XE seattle versions I am using, more investigation...
I have seen enough, it's definetly a bug in Delphi XE Seattle, 99% sure it's inside the compiler. 1% it may have to do with code/error insight not going to test this any further and for now use Delphi XE7 to do re-structuring.
There are more circular references when trying to restructure for example TPCBank and TOperationComp.
Even Delphi XE7 will run out of memory, so I am starting to suspect that no Delphi compiler version is 100% capable of dealing with this, though this situation may have progressed from minor to worse over many XE versions.
Perhaps earlier Delphi versions will not run out of memory so fast. One possible cause could be the RTTI which is added everywhere in newer XE versions... hmmm...
One possible solution might be using "interfaces" perhaps without referencing counting to be able to implement certain behaviour without causing circular references ?!? Not sure if using interfaces would solve it... but perhaps... I don't like interfaces though, so a different solution will have to be found if restructuring is going to be completely successfull.
I think a better solution might be too seperate functionality more into as much as possible isolated units, which can then be re-used more where necessary, this should break circular references.
(Emit RTTI compiler option seems to be off, so this hypothesis that RTTI may worsen the situation seems wrong, though perhaps this option is "legacy" not sure ;))
(I tried using the MSBuild trick, this does "spawn" a dcc32.exe compiler process or something like that and it simply goes to 2GB and then terminates, so it's clearly a compiler problem/bug, so this trick doesn't help, also this trick has a nasty side effect/consequences, files will have to be saved first, which is a very nasty requirement).
Hmm it's kinda interesting to use Free Pascal/Lazarus and see how far it can get. I can already see that it produces different kinds of warnings then Delphi, somewhat usefull.
FPC did find some uninitialize fields in the const null structures and also a variable vs function conflict with same name, though Delphi will allow it and will handle it ok, it is better to avoid variables and functions with same name so FPC was somewhat usefull here ;) May try FPC/Lazarus some more tomorrow, very maybe. So far it's not producing any circular reference problems, but this is because it's choking on code it can handle, once those errors are resolved it will be interesting to see if FPC is better or worse at detecting circular problems =D
Some of these errors because I did not yet set {$MODE Delphi} in project file, this should solve some errors, will try again and see how that goes ;) Probably should have set FPC define or so, need to figure out how to add defines to lazarus project, so far don't know but managed to get everything work with some little tweaks here and there.
So far conclusion, fpc/lazarus 0.9.30 can detect these circular situations without going out of memory (for now), so this is a bit interesting :)
fpc/lazarus 1.6.4 can also detect it without problems so far.
Further restructuring advise:
For example (objects/units) A uses B uses C uses A.
Advise is to "pull out" routines/methods which call into each other and place those routines in a new object/unit D.
The idea is to completely isolate A, B, C from each other and instead create a new object D which can safely uses A,B,C... and pass data from A,B,C via D.
Restructering is possible with git rebase too, this may update changes from original into a re-structured version though this is probably a little bit less ideal, but I may try this just to see how it works. Though perhaps it's better to keep the structure as it is right now, though I would have much like some restructuring to split off non-essential classes/helper classes into seperate units.
Currently the new folder structures proved a problem though, this will need special processing. One idea could be to create "temp" gits/repositories to fix that up, not sure about exact methods used... some developers use scripts for this, quite complex operation, to complex for me for now, may look into this some more in near future.
After some GIT research/experiments/investigations, the assumptions are as follows:
GIT is incapable of recognizing big file split into multiple files. (Though once it recognized a two-way split, have not been able to reproduce this recognization.)
GIT is even worse when these split files are moved.
GIT is incapable of updating the splitted files if the older big file is updated.
Perhaps in the far future when GIT becomes more advanced GIT might be capable of recognizing such cases. For now I will assume GIT is not capable of this functionality.
To study/research/learn how PascalCoin works it is necessary to split the big files into multiple little files, this will make it easier to spot and isolate functionality. This will allow to analyze pascalcoin based on "functionality parts".
Since GIT history is completely useless anyway in this regards I will probably remove the git history for this "restructured project". I will however put a little note in to tell on which commits of PascalCoin it was based and output a git log to a file to be able to tell it's history at a glance so that even if hashes are changed the commit history should be enough to figure out on which versions/commits it was based.
To keep GIT history as clean as possible for this restructured project further alterations may be decided upon:
Under consideration is:
1. Removal of Delphi GUI application. (Could be seperated into it's own GIT repository)
2. Removal of Free Pascal/Lazarus GUI application. (Could be seperated into it's own GIT repository)
3. Removal of Free Pascal un-used units/folders.
Additions:
4. Create console based test programs to test/compile/build "core" of pascal coin.
5. Create console based test programs to test only specific parts of pascal coin.
These could also be seperated into their own GIT repository.
At risk is code compatibility if seperation is to be done. One possible solution is to include versioning into unit filenames, though this will quickly become fatigueing/obnoxious and kinda defeats the purpose of GIT/a versioning system. Though GIT does allow multi user and branching/evolutionary approach and such.
Given the complexity of the project seperation may have to be done anyway to keep it somewhat managable and extendeable in the future and keep it a bit clean.
One possible/better solution is to clearly state which version of which repository the code was written against. Perhap the hash of the commit could be used to indicate this, though hashes can be delete/removed/altered etc, git history can be altered as well so it's no perfect garantuee, so perhaps hashes are to be avoided, git itself may have to be avoided for these purposes but for now it's a nice tool to use.
Repository split-up conceptual example:
PascalCoinCore
PascalCoinDelphiGUI
PascalCoinLazarusGUI
PascalCoinLazarusMiner
PascalCoinCoreTests
Core should remain buildable by Delphi and Lazarus/FreePascal.
PascalCoinDelphiGUI could be isolated to Delphi compilers. Though perhaps Lazarus can build it too.
PascalCoinLazarusGUI can be isolated to Lazarus.
PascalCoinLazarusMiner is for compiling with Lazarus.
PascalCoinCoreTests is for test programs testing the core of PascalCoin.
One disadventage is if LazarusMiner becomes buildable by Delphi then the repository name is a bit strange. Though a new repository could be created for a DelphiMiner. Eventually shared code could be split into PascalCoinMinerCore.
This will allow maximimum flexibility and customization.
Though in the future it may become even more complex as different platforms might be added.
(Is PascalCoin for android open source ? :) where is it ? =D)
Anyway the re-structured PascalCoinRestructured repository will be available soon. It will still include the Delphi GUI and Lazarus GUI and Lazarus Miner and basically all other files. So that it is a complete distribution and no files were deleted.
After which new repositories can be created to mimic the re-structured project and to split it off into multiple repositories.
For this I have already created these repositories and will fill them soon with code =D
This idea has some big advantages:
1. PascalCoinCore can be build into "packages", which are special DLLs and can also generate files suited for C/C++. This will make it easier to integrate PascalCoinCore into C/C++ projects.
2. Further adventage is when PascalCoinCore changes only the DLL has to be replaced as long as GUI stays the same and vice versa, this can cut back on ammount of binaries generated and safe lots of space in repositories.
3. Ultimately once in a while everything can be integrated into one BIG exe so that functionality is not lost.
I am currently investigating the possibilities that Delphi XE offers and to see if and how well this works. So far it seems to require simply to specific "." as output folders to get these files produced. Further investigation into this is under way =D
Unfortunately using "Delphi Package" has some major drawbacks. All "type" info is lost, methods/properties etc, there are some solutions like using RTTI/published properties or COM/interfaces, but this is all way too complex. This is for "dynamically linked packages" *explicitly* loaded at runtime.
There is something else though, "link with runtime packages" this seems to do a similiar thing but here all type info can be accessed via DCP file, and later it will load BPLs and thus BPLs can be replaced with different versions, so this is definetly an interesting but confusing feature ! ;)
(Seem similiar as to above but with massive automation via DCPs or so, except applications cannot handle failure to load BPLs elegantly or flexibly thus users may end up in the dark trying to figure out why there application with missing BPL is not working or loading the wrong version.)
It also requires the files to be present in certain folders otherwise Delphi cannot find it which is unfortunate, this should have been simplified by using "Add to Project', and simply adding DCPs or BPLs to project manager to be able to tell which ones to use or at least where to find the DCP... now it's all a bit confusing via project options->packages->runtime packages. No real way to tell which DCP version is being used, this is very dangerous and unfortunate and will absolutely lead to MAXIMUM confusion.
However with some carefull instructions perhaps in readme files users may be informed to always update this entry with latest DCP file if they are to be MAX compatible with latest PascalCoinCore BPL and such.
Kinda interesting approach may try this with PascalCoinCore to see how much confusion is added by not seeing where the files reside... but the adventages are kinda cool... being able to swap BPLs with different versions :)
There is more potential for confusion though, if same BPL is already on a system path it may load that instead of none is found in local folder. Plus windows 10 has major loading issues with BPLs.
This may be fixed in later Delphi XE compiler versions... perhaps it's already fixed, not sure which Delphi version has these windows 10 fixes.
http://blog.marcocantu.com/blog/2017-june-delphi-packages-creators-update.html
This is a handy little tool it can show which DLLs are loaded and from where, also works for BPLs, this can solve some of the DLL HELL of not knowing where DLLs are loaded from (it shows full paths unlike Delphi IDE/debugger which only shows package name):
https://docs.microsoft.com/en-us/sysinternals/downloads/listdlls
Usage:
"
Parameter | Description
processname | Dump DLLs loaded by process (partial name accepted).
pid | Dump DLLs associated with the specified process id.
dllname | Show only processes that have loaded the specified DLL.
-r | Flag DLLs that relocated because they are not loaded at their base address.
-u | Only list unsigned DLLs.
-v | Show DLL version information.
"
Delphi does something stupid/annoying, it adds these kind of folders to the windows path environment variable:
"C:\Users\Public\Documents\Embarcadero\Studio\17.0\Bpl;"
and default it wil generate/build BPLs into these somewhat unknown folders, confusing users as somehow these BPLs are "magically" loaded from unknown locations, this one above.
The little tool can shed some light on this. I have multiple of these folders in my path variable, for now I will let them be since some Delphi installations and special components may rely on these folders and the BPLs located there.
I wish this was not the case and Delphi would avoid polluting my system path variables with these silly BPL folders.
Quite dangerous too, suppose my applications use these BPLs without me knowing about it and then later Delphi is removed/uninstalled from system, and/or applications moved to different systems they would stop/seize functioning if they linked to see BPLs in some way, usually this requires some special checkboxes to be checked though, otherwise it will be statically/compiled into the EXE so I believe.
Still when experimenting with "linked" and "loading" this will very quickly become a source of confusion.
I am a fan of "explicitly" telling IDEs where to load stuff from and also partial paths and such, much more clear.
In future I will try and avoid these BPL folders and instead store them in project settings/search paths that kind of thing, hoping to keep my path variable clean and to avoid any possible confusion and link issues.
Unfortunately preventing confusion on my system will not be enough, other Delphi users will continue to experience confusion when trying to use BPLs. Only readme files and such could try and solve some of that confusion.