78 Commits

Author SHA1 Message Date
Abhishek Kondur edda2868fc NM-163: Users, Groups, Roles, Networks and Hosts Table Migration (#3910)
* feat(go): add user schema;

* feat(go): migrate to user schema;

* feat(go): add audit fields;

* feat(go): remove unused fields from the network model;

* feat(go): add network schema;

* feat(go): migrate to network schema;

* refactor(go): add comment to clarify migration logic;

* fix(go): test failures;

* fix(go): test failures;

* feat(go): change membership table to store memberships at all scopes;

* feat(go): add schema for access grants;

* feat(go): remove nameservers from new networks table; ensure db passed for schema functions;

* feat(go): set max conns for sqlite to 1;

* fix(go): issues updating user account status;

* refactor(go): remove converters and access grants;

* refactor(go): add json tags in schema models;

* refactor(go): rename file to migrate_v1_6_0.go;

* refactor(go): add user groups and user roles tables; use schema tables;

* refactor(go): inline get and list from schema package;

* refactor(go): inline get network and list users from schema package;

* fix(go): staticcheck issues;

* fix(go): remove test not in use; fix test case;

* fix(go): validate network;

* fix(go): resolve static checks;

* fix(go): new models errors;

* fix(go): test errors;

* fix(go): handle no records;

* fix(go): add validations for user object;

* fix(go): set correct extclient status;

* fix(go): test error;

* feat(go): make schema the base package;

* feat(go): add host schema;

* feat(go): use schema host everywhere;

* feat(go): inline get host, list hosts and delete host;

* feat(go): use non-ptr value;

* feat(go): use save to upsert all fields;

* feat(go): use save to upsert all fields;

* feat(go): save turn endpoint as string;

* feat(go): check for gorm error record not found;

* fix(go): test failures;

* fix(go): update all network fields;

* fix(go): update all network fields;

* feat(go): add paginated list networks api;

* feat(go): add paginated list users api;

* feat(go): add paginated list hosts api;

* feat(go): add pagination to list groups api;

* fix(go): comment;

* fix(go): implement marshal and unmarshal text for custom types;

* fix(go): implement marshal and unmarshal json for custom types;

* fix(go): just use the old model for unmarshalling;

* fix(go): implement marshal and unmarshal json for custom types;

* feat(go): remove paginated list networks api;

* feat(go): use custom paginated response object;

* fix(go): ensure default values for page and per_page are used when not passed;

* fix(go): rename v1.6.0 to v1.5.1;

* fix(go): check for gorm.ErrRecordNotFound instead of database.IsEmptyRecord;

* fix(go): use host id, not pending host id;

* feat(go): add filters to paginated apis;

* feat(go): add filters to paginated apis;

* feat(go): remove check for max username length;

* feat(go): add filters to count as well;

* feat(go): use library to check email address validity;

* feat(go): ignore pagination if params not passed;

* fix(go): pagination issues;

* fix(go): check exists before using;

* fix(go): remove debug log;

* fix(go): use gorm err record not found;

* fix(go): use gorm err record not found;

* fix(go): use user principal name when creating pending user;

* fix(go): use schema package for consts;

* fix(go): prevent disabling superadmin user;

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* fix(go): swap is admin and is superadmin;

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* fix(go): remove dead code block;

https://github.com/gravitl/netmaker/pull/3910#discussion_r2928837937

* fix(go): incorrect message when trying to disable self;

https://github.com/gravitl/netmaker/pull/3910#discussion_r2928837934

* fix(go): use correct header;

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* fix(go): return after error response;

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* fix(go): use correct order of params;

https://github.com/gravitl/netmaker/pull/3910#discussion_r2929593036

* fix(go): set default values for page and page size; use v2 instead of /list;

* Update logic/auth.go

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* Update schema/user_roles.go

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* fix(go): syntax error;

* fix(go): set default values when page and per_page are not passed or 0;

* fix(go): use uuid.parse instead of uuid.must parse;

* fix(go): review errors;

* fix(go): review errors;

* Update controllers/user.go

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* Update controllers/user.go

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* NM-163: fix errors:

* Update db/types/options.go

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* fix(go): persist return user in event;

* Update db/types/options.go

Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>

* NM-163: duplicate lines of code

* NM-163: fix(go): fix missing return and filter parsing in user controller

- Add missing return after error response in updateUserAccountStatus
  to prevent double-response and spurious ext-client side-effects
- Use switch statements in listUsers to skip unrecognized
  account_status and mfa_status filter values

* fix(go): check for both min and max page size;

* fix(go): enclose transfer superadmin in transaction;

* fix(go): review errors;

* fix(go): remove free tier checks;

* fix(go): review fixes;

---------

Co-authored-by: VishalDalwadi <dalwadivishal26@gmail.com>
Co-authored-by: Vishal Dalwadi <51291657+VishalDalwadi@users.noreply.github.com>
Co-authored-by: tenki-reviewer[bot] <262613592+tenki-reviewer[bot]@users.noreply.github.com>
2026-03-17 19:36:52 +05:30
Vishal Dalwadi a4981ffd26 NM-168: Network Flow Logs (#3754)
* feat(go): define flow events;

* feat(go): improve structure;

* feat(go): improve structure;

* feat(go): remove old flow definitions;

* feat(sql): add clickhouse init scripts;

* feat(sql): add protobuf spec;

* fix(sql): store ip as string;

* feat(go): move proto def to grpc dir;

* feat(go): use node instead of host as type; optimize protobuf defs;

* feat(go): add clickhouse db support; add endpoint to query flows;

* fix(go): fix clickhouse config;

* fix(go): use error response structure to report error;

* feat(go): pass flow logging status to netclient;

* feat(go): add peer ip identity map to host peer info;

* feat(go): remove prefix from participant obj fields;

* feat(go): add flow logs enabled field to host;

* feat(go): add filtering to get flow api;

* feat(go): fix record struct;

* feat(go): add exporter url to server config;

* feat(go): add exporter url to server config;

* feat(go): enable flow logs by default;

* feat(go): update nm-quick.sh;

* feat(go): update nm-quick.sh;

* feat(go): update nm-quick.sh;

* feat(go): update nm-quick.sh;

* feat(go): add db initialization logic;

* feat(go): filter by network id;

* fix(go): connection issue;

* fix(go): connection issue;

* fix(go): golang builder version;

* feat(go): add server settings for flow logs;

* feat(go): initialize clickhouse in pro; check for retention;

* feat(go): add exporter feature flags;

* feat(go): add grpc behind caddy;

* feat(go): expose ports correctly;

* fix(go): grpc caddyfile config;

* fix(go): publish exporter feature flags on license validation;

* fix(go): set server name for netmaker exporter;

* fix(go): set server name for netmaker exporter;

* fix(go): check for nil cancel func;

* fix(go): add flow logs field to api host;

* fix(go): add flow logs field to api host;

* fix(go): remove port from grpc setting;

* chore(go): tabs;

* feat(go): introduce egress range participant type;.

* feat(go): rename egress range to egress route for uniform language;

* feat(go): rename egress range to egress route for uniform language;

* feat: add peer addr identity map to host peer update;

* feat: add address identity map to host peer update;

* feat: add address identity map to host peer update;

* feat: set correct from and to args;

* feat: add support for filtering by node;

* feat: use corresponding base image;

* feat: update dockerfile base image version;

* fix: disable flow logs for all host when global settings are changed;
2025-12-12 14:12:00 +04:00
Abhishek Kondur eed32cd2d6 Merge pull request #3735 from gravitl/NM-166
NM-166: Device Posture Checks
2025-12-05 10:33:11 +04:00
Abhishek K c643a50b67 NM-159: simplify auto assign gw logic (#3719)
* force update host dns field if node is acting as inet gw

* add old acl support checks

* move auto relay migration to pro pkg

* add check to avoid unsetting relayed node

* simplify auto assign gw logic

* send auto assign update on un relay

* set checking time to latest on updates

* fix HA auto Relay logic

* add relay node metrics to peer signal

* move auto relay peer check

* publish host peer update

* check and unset unrelayed auto peers

* use relay node mutex to avoid rac condition

* reset autorelayed peers on auto assign gw
2025-11-06 14:54:09 +04:00
Abhishek K 74fef9fbc6 NM-122: Auto Relay, auto assignment of Gw (#3697)
* add auto realy handlers and logic funcs

* add pro func connectors

* Add auto relayed peer ips on peer update, set auto relay on gw creation

* add network id to signal, add autorelay nodes to peerudpate

* add autorelay peer update logic

* add nodes to peer update

* revert node model change

* reset auto relayed peers on the relay node on reset, add auto relay nodes to pull

* add logic api to update auto relay node

* add autoassigngw field to node, add logic to swith relay node in relayme udpate api

* add gw nodes to pull

* intilaise gw map

* HA relay functionality

* add autoassign gw option to enrollment key

* publish intant action to auto assign gw

* fix static checks

* unset relay if auto assign removed

* add host node model to auto relay info

* add host node model to auto relay info

* only use hostNode model for gws info

* handle autoassigned gw peer in the update

* handle autoassigned gw peer in the update

* handle peer updates for autoassigned gw peer

* unset auto assigned peer if relayed or failedovered
2025-10-28 09:53:31 +04:00
Abhishek K 49e28e3385 NM-137: Add addtional mq actions to host api (#3671)
* add host node update action

* add peer signal action to fallback api

* add replace peers to host pull

* add delete host action to fallback api

* update base go builder image

* update go builder tag

* check host port to avoid conflicts behind NAT

* fix connect/disconnect on api

* send pull signal on disconnect from UI

* fix panic on host join via user auth

* reset failover on disconnect
2025-10-07 13:16:31 +04:00
Abhishek K 9e0196126f NM-79: Domain Based Egress Routing (#3607)
* add support for egress domain routing

* add domain info to egress range

* fix egress domain update

* send peer update domain resolution update

* add egress domain update in the peer update

* use range field for domain check

* add egress domain to host pull

* add egress domain model to egress host update

* add egress domain model to egress host update

* update egress domain model on acls

* add check of range if domain is set

* sync egress domains to dns system

* add egress domain to match domain list, fix egress nat rule for domains

* fix all rsrcs comms

* fix static checks

* fix egress acls on CE

* check for all resources access on a node

* simplify egress acl rules

* merged ce and pro acl rule func

* fix uni direction acl rule for static nodes

* allow relayed nodes traffic

* resolve merge conflicts

* remove anywhere dst rule on user node acls

* fix: broadcast  user groups update for acl changes

* add egress domain ans routes to nodes

* add egress ranges to DST

* add all egress ranges for all resources

* fix DNS routing acls rules
2025-09-11 15:24:17 +05:30
abhishek9686 d978de08d0 collect host localtion for graph 2025-06-12 15:47:24 +05:30
Abhishek K 5849e0e2e1 NET-1932: sort out acls into CE and PRO (#3460)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-02 13:05:10 +05:30
Vishal Dalwadi cec48be354 NET-1906: Add endpoints for Sync All Hosts and Upgrade All Hosts. (#3302)
* feat(go): add endpoint to sync all hosts.

* feat(go): add endpoint to upgrade all hosts.

* feat(go): allow force upgrade of hosts.

* fix(go): config yaml tag.
2025-01-28 11:33:38 +05:30
Yabin Ma 1d9e0f71df add format int64 for swagger doc (#3247)
* add format int64 for swagger doc

* add format int64 for swagger doc

* add global override

* swagger doc update

* add new line for .swaggo file
2024-12-12 09:52:52 +04:00
Max Ma 65faf73fe9 NET-1226: Scalability Improvements (#2987)
* add api to check if failover node existed

* remove 5 minute peerUpdate

* update peerUpdate to trigger pull

* update Action name to SignalPull

* revert the peerUpdate from SignalPull

* fix getfailover error issue

* rm acls creation for on-prem emqx

* remove use of acls

* add additional broker status field on status api

* NET-1165: Remove creation of acls on emqx (#2996)

* rm acls creation for on-prem emqx

* remove use of acls

* add additional broker status field on status api

* comment out mq reconnect logic

* configure mq conn params

* add metric_interval in ENV for publishing metrics

* add metric_interval in ENV for publishing metrics

* update PUBLISH_METRIC_INTERVAL env name

* revert the mq setttings back

* fix error nil issue

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
Co-authored-by: Abhishek K <32607604+abhishek9686@users.noreply.github.com>
2024-07-09 18:56:55 +05:30
Max Ma c7469c79d0 rename isstaticendpoint to isstatic 2024-06-03 10:37:19 +02:00
Max Ma 81a6c9cf8a seperate static port and static endpoint 2024-05-28 10:16:11 +02:00
Max Ma 5740c3e009 Net 1115 (#2890)
* add endpointipv6 for host

* keep endpointipv6 unchanged when enable static endpoint

* handle ipv6 endpoint updates

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
2024-04-11 17:37:45 +05:30
Abhishek K 465f2bd5be NET-896: Scale test bug fixes (#2764)
* send peer update in async

* update metrics on fallback

* return http json response
2024-01-15 23:17:36 +05:30
Abhishek K 98c01c4325 NET-814: Deprecating TURN (#2723)
* deprecate turn

* process signals through mq
2023-12-13 22:46:57 +04:00
Abhishek K b78cc0a8a1 NET-725: Failovers (#2685)
* api to  to get host relayed from client

* add auto relay to api host

* add peer nat type

* set pro field on signal

* rm net check on relay me handler

* return success response

* re-establish failover logic

* set failOver ctx

* failOver with peer pub key

* failovered peer updates

* failover handlers, reset failovered peer on deletion

* rm unused funcs

* initialize failover handler on EE

* ignore failover node on signal

* failover changes

* set host id on signal

* extend signal model to include node ids

* add backwards compatibility

* add failover as node api

* set json response on failover handers

* add failover field to api node

* fix signal data check

* initialize failover peer map

* reset failovered status when relayed or deleted

* add failover info to api node

* reset network failover

* only proceed furtuer if failover exists in the network

* set failOver node defaults

* cannot set failover node as relayed

* debug log

* debug log

* debug changes

* debug changes

* debug changes

* revert debug changes

* don't add peers to idmap when removed

* reset failed Over

* fix static checks

* rm debug log

* add check for linux host
2023-11-29 20:10:07 +04:00
Aceix 61ef6142ff feat(NET-688): auto relaying via enrollment keys (#2647)
* feat(NET-688): auto relaying via enrollment keys

* feat(NET-688): address pr comments
2023-11-04 16:28:57 +04:00
Gabriel de Souza Seibel cb4b99ffcb [NET-562] Persistent Keep Alive from node to host (#2604)
* Move PKA field from models node to host level

* Move PKA field from api models node to host level

* Adapt logic package to node->host PKA

* Adapt migration-related code to node->host PKA

* Adapt cli code to node->host PKA

* Change host PKA default to 20s

* On IfaceDelta, check for PKA on host

* On handleHostRegister, set default PKA

* Use a default PKA

* Use int64 for api host pka

* Reorder imports

* Don't use host pka in iface delta

* Fix ConvertAPIHostToNMHost

* Add swagger doc for host PKA field

* Fix swagger.yml

* Set default PKA only for new hosts

* Remove TODO comment

* Remove redundant check

* Have api-host pka be specified in seconds
2023-10-06 10:09:19 +04:00
Gabriel de Souza Seibel 23ad85bd8c Add handler that sends MQ msg to upgrade a host (#2582) 2023-10-02 10:37:31 +04:00
Aceix 596cc7a945 feat(NET-449): add sync feature to request a host pull from server (#2491) 2023-08-04 13:22:47 +05:30
Aceix 91859c2490 prune(NET-483): remove defunct host.internetgateway field (#2487) 2023-08-02 17:26:44 +05:30
Abhishek K ab73278597 add timestamp to turn signal (#2470) 2023-07-28 09:20:43 +05:30
Abhishek K 645bc4c126 NET-447: Removed proxy related fields and code (#2459)
* remove related fields and code

* remover metrics collection from server code

* fw update struct

* add ext client flag to metrics data

* simply nat types

* rm proxy update from cli
2023-07-26 11:22:49 +05:30
Abhishek Kondur 243cc74490 remove endpoint detection from host model 2023-06-27 22:36:24 +05:30
Tobias Cudnik 94ca72e370 NET-173 Sync EndpointDetection config (#2374)
* adding serverconfig logic and variable to hosts

* - sync EndpointDetection on peerupdate
- add EndpointDetection to env config

* - endpoint detection always comes from the server
- fixed ENDPOINT_DETECTION -> NETCLIENT_ENDPOINT_DETECTION

---------

Co-authored-by: afeiszli <alex.feiszli@gmail.com>
2023-06-13 16:33:53 -04:00
Abhishek K 47edf65b1f NET-186: Wg public listen port (#2344)
* fetch public listen of wg if present

* check if wg pub listen port has been changed on host update

* wg public port to host api model for visibility

* rm comment
2023-05-31 08:21:02 +05:30
Abhishek K 46acf1d336 NET-129: Turn Signal Actions (#2290)
* add signal action field

* add negotiation signal action

* typo fix

* change signal action name
2023-05-15 15:40:34 -04:00
Abhishek Kondur 9b87a6b722 remove unused fields 2023-04-26 17:03:25 +04:00
Abhishek Kondur 9e0038f39c merge conflicts resolved 2023-04-25 11:12:22 +04:00
Matthew R Kasun 754ab44a48 Merge remote-tracking branch 'origin/develop' into GRA-1529-key-update 2023-04-19 09:48:04 -04:00
dcarns 3d82df3bc6 Merge pull request #2133 from gravitl/refactor_auto_update
Add host level auto update setting with default true
2023-04-19 09:40:56 -04:00
Abhishek Kondur 433b61417b merge conflicts resolved 2023-04-19 13:23:06 +04:00
Abhishek Kondur aaa2211939 add server field to signal struct 2023-04-19 07:05:17 +04:00
0xdcarns 18c9bcc58f adapted sso to host registration 2023-04-17 22:23:17 -04:00
Abhishek Kondur d4ceabd4b3 register host through server, add basic auth to turn apis 2023-04-17 11:21:28 +04:00
Matthew R Kasun 63552891ee refactor key updates on host basis vice node 2023-04-13 15:02:14 -04:00
Abhishek Kondur 1324b6443d add reply field to signal struct 2023-04-13 10:00:09 +04:00
Abhishek Kondur 726531af25 change relayendpoint type to str 2023-04-11 09:46:51 +04:00
Abhishek Kondur 4d223c1cdc api for signalling peer 2023-04-10 12:14:40 +04:00
Abhishek Kondur f54ae9ae8f turnserver registration apis 2023-04-06 08:50:34 +04:00
Abhishek Kondur b8904d1f3c Merge branch 'develop' of https://github.com/gravitl/netmaker into GRA-1494 2023-04-04 18:32:55 +04:00
Abhishek Kondur 5e8e56b56e merge conflicts resolved 2023-04-03 22:43:33 +04:00
0xdcarns fa40a5929d added turn ep + nat type to host model 2023-03-30 17:23:21 -04:00
Abhishek Kondur 851fdbfee1 add turn config to host model 2023-03-22 12:05:51 +04:00
0xdcarns bf687dd478 add return 2023-03-20 08:55:28 -04:00
Anish Mukherjee 6b684d95a4 add host auto updated setting with default true 2023-03-20 17:47:41 +05:30
afeiszli 0bbf1dbe54 hotfixing logging and model issues found during IoT testing 2023-03-16 13:41:56 -04:00
Matthew R Kasun 8b5d194666 Merge remote-tracking branch 'origin/develop' into GRA-1375-default-proxy 2023-03-15 13:24:52 -04:00